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ABSTRACT 

Research was conducted between 1996 and 1998 with the aim of 
developing a methodology for assessing and monitoring children's learning in 
developing countries. The study was framed by the multi-disciplinary 
theoretical framework of socio-cultural research, drawing specifically on the 
methodological orientation of action and intervention in research. In this 
case, 30 teachers each in Malawi and Sri Lanka took part in a system of 
activities in which they developed appropriate protocols for classroom-based 
assessment, designed and administered a variety of novel assessment tasks, 
collected evidence of children’s work and discussed these collectively in 
groups, and recorded and profiled children's achievement in literacy and 
mathematics. Results indicated that in both countries teachers had reasonably 
good intuitive knowledge of students and learning but inadequate systems for 
collecting and interpreting evidence of achievement in literacy and 
mathematics. Once they were exposed to the intervention process of the 
research, it became clear that teachers, despite enormous resource 
constraints, were able to establish standards and norms based on their 
professional experiences, develop procedures for assessment, administer 
assessment tasks, and develop portfolios of evidence and assemble a profile 
of student achievements. Challenges to achieving this are not be 
underestimated and do vary between socio-cultural contexts. Four sets of 
profiles were developed in both English and the national languages of the 
countries. These were: (1) oracy profiles; (2) reading profiles; (3) writing 

profiles; and (4) mathematical profiles. The study concludes that the 
activity-based intervention is a useful approach to developing insights into 
learning, teaching , and monitoring educational quality. (Contains 44 
references and 21 figures.) (Author/BT) 
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The research was carried out between 1996 and 1998 with the aim of developing a methodology 
for assessing and monitoring children’s learning in developing countries. 

The study was framed by the multi-disciplinary theoretical framework of socio-cultural 
research, and drew specifically on the methodological orientation of ‘action’ and ‘intervention’ 
in research. 

In this case, 30 teachers in each country (Malawi and Sri Lanka) took part in a ‘system of 
activities’ in which they developed appropriate protocols for classroom-based assessment, 
designed and administered a variety of novel assessment tasks, collected evidence of children’s 
work and discussed these collectively in groups, and recorded and profiled children’s 
achievements in Mathematics and literacy. 

The study found that in both countries teachers had reasonably good intuitive knowledge of 
learners and learning, but inadequate systems for collecting and interpreting evidence of 
achievement in literacy and Mathematics. 

Once they were exposed to the ‘intervention 4 process of the research, it became clear that 
teachers, despite enormous resource constraints, were able to establish standards and norms 
based on their professional experiences, develop procedures for assessment, administer 
assessment tasks and put together portfolios of evidence and assemble a profile of learner 
achievements. 



The challenges to achieving this should not be under-estimated and do, of course, vary between 
socio-cultural contexts. 



Four sets of profiles were developed in both English and the national languages of the different 
countries. These were: 



Oracy profiles 

Reading profiles 
Writing profiles 
Mathematics profiles 



(English and Chichewa in Malawi; English, Sinhala and Tamil in Sri 
Lanka) 

(as above) 

(as above) 

(Pupils in Malawi were assessed in English and in Sri Lanka they were 
assessed in Sinhala and Tamil) 



In Malawi, the oracy profiles show that 27% of children achieve the level expected by their 
teachers in English and 40.7% achieve the expected level in Chichewa. 

In Sri Lanka, 20.3% of children achieved the expected level in English, 20% did so in Tamil and 
20.7% in Sinhala. 



The assessment of reading in Malawi shows that 32.4% of children achieved the expected level 
of reading texts in English and 34.4% in the reading of texts in Chichewa. 

In Sri Lanka, the teacher-based expectation that at least 50% of children would achieve a level 3 
in the reading of English, Sinhala and Tamil texts was not borne out. 

The writing profiles of children in Malawi are strong, with 40.5% achieving a level 3 in English 
and 42.8% in Chichewa. 

By contrast, the writing profiles in Sri Lanka were weak in English. Only 2.5% of children 
achieved a level 3, but this was unsurprising as the pattern of teaching English is, in the first 
instance, to develop the communicative competencies in speaking and reading. 

However, the profiles for writing in Sinhala and Tamil were more encouraging, though falling 
below teacher expectations (only 39% achieved a level 3 in Sinhala and 38.7% in Tamil). 

The Mathematics profile, in both countries, shows that under 40% of learners achieve a level 3. 
In general, learners, in both countries, showed a high level of success in computational problems 
in which ‘rules’ were used, but were less successful in word-based problem-solving, diagnosis 
and measurement. 



The study concludes that the ‘activity-based intervention’ methodology is a useful approach to 
developing insights into learning and teaching, and for monitoring education quality. 



Chapter 1 - Introduction 
Rationale for the study 

There is a paucity of research into children's learning in developing countries. The studies which do 
exist, mostly describe the quality of an educational system, or part of the system, in terms of 'input' 
into the teaching process (teachers, equipment, materials, etc.) or look at student achievement in 
relation to these inputs (Jansen, 1995; Lockheed and Verspoor, 1991). Such studies are often 
conducted outside the real context of the classroom and are largely summative. There is very little 
research which uses 'insider perspectives' (Prophet & Rowell, 1990; Sato, 1990) of what happens in 
schools and classrooms, nor are there many studies which have adopted approaches which were able to 
monitor standards over time. 

Moreover, although teachers are the most natural 'insiders' and indeed carry out assessments into 
children's learning, the methods of assessment, recording and interpreting learning outcomes they 
currently use rarely provide adequate information upon which to make judgements about the quality of 
education. It was to address these two needs that this study into the quality of learning and teaching in 
two developing countries was undertaken. 

Aims of the Study 

The main aim of the study was to develop a framework in which the achievements of primary aged 
children in Malawi and Sri Lanka in Literacy and Mathematics could be described. This objective was 
to be achieved through a methodology which involved developing appropriate protocols for classroom- 
based assessments, collecting evidence of children’s work, and recording their achievements in terms of 
literacy and numeracy ‘profiles’. 

The specific aims of the study were 

i. to develop a profiling system capable of providing reliable and valid indicators of children's 
learning at particular stages of primary schooling 

ii. to enhance the capacity of teachers to collect information about children’s learning and to keep 
records of their achievements. 

Method and implementation 

The research fell broadly within the tradition of developmental and 'interventionist' approaches 
encouraged by socio-cultural and activity theorists (Cole 1995; 1998; Wertsch et al 1995). The 
methodological approach adopted was based on the belief that socio-cultural studies should be involved 
in changing, and not just examining, human action and the cultural, institutional and historical 
settings in which it occurs (Wertsch, del Rio and Alvarez 1995:29). 

Thus, the research approach adopted in this study set out to bring about changes to learning and 
teaching through encouraging teacher and pupil activity. The ultimate gaol was to achieve 'sustainable 
new systems of activity in existing institutions', (Cole 1995:199) 

There were five broad stages of the research process. 30 teachers in each country were involved in: 

i. Reviewing the learning requirements as stated in curricula, syllabi, and textbooks in each country 
for particular stages of primary schooling; 

ii. Establishing standards and norms based on their professional experiences and an emerging 
literature on learning; 

iii. Developing procedures for assessment; 

iv. Administering assessment tasks, collecting evidence of children’s achievements and recording 
outcomes; 

v. Developing profiles of children’s achievements 
The stages are described in more detail in chapter 4. 



Chapter 2 - The Quality of Education in Developing Countries - a review of the 
literature 

Introduction 

'Education Quality’ as a concept has always been difficult to define. Most public debates on the quality 
of education include concerns about a student's level of achievement, the relevance of learning to the 
world of employment or the social, cultural and political worlds occupied by the student Frequently 
they often also include concerns about the conditions of learning, such as supply of teachers or 
facilities. 

In the light of this, researchers have suggested that the concept of educational quality is complex and 
multidimensional (Grisay & Mahlck, 1991; Hawes & Stephens, 1990 ). Grisay and Mahlck (1991) 
argue that the notion of quality should not be limited to student results alone but should also take into 
account the determinant factors which influence these, such as the provision of teachers, buildings, 
equipment, curriculum and so on. As such, the general concept of quality of education is made up of 
three interrelated dimensions. These are: the quality of human and material resources available for 
teaching (inputs), the quality of teaching practices (process) and the quality of results (outputs and 
outcomes). 

Thus, studies which set out to assess the quality of education need to be treated carefully. Some 
studies which purport to assess the quality of education are in fact simple measures of input to 
education (teachers, equipment, materials, etc.) Many of these studies are problematic because they 
focus on formal rather than actual quality characteristics. For example, one school might have a larger 
number of highly qualified teachers than another, but they may be less motivated. Similarly, one 
school might have fewer facilities than another, but use them more efficiently (Carron & Ta Ngoc, 
1981). 

Another set of studies are those which use indicators such as repetition rates and drop-out rates as 
proxy measures of educational quality. The attractiveness of such studies is the availability of data, 
often contained in educational statistics collected through Educational Management Information 
Systems in most developing countries. According to Lockheed and Hanushek (1987), these data are 
useful for making aggregate comparisons between regions of a country, and between countries, but are 
less relevant for analysing differences in performance between schools and between children in the same 
grade. They are even less useful for explaining such differences (Alexander et al, 1999). 

Many studies do collect data on student achievement. However, most such data are based on standard 
achievement tests and tend to focus on the acquisition of traditional knowledge and skills. According 
to Ross and Mahlck (1990), the attainment of more complex educational objectives, such as 
'individuals capable of working in co-operation with others' or 'demonstrating ability to solve 
problems' are rarely evaluated. Indeed, looking at student outcomes alone does not tell us how schools 
operate. A school whose students achieve a higher score than those of another is not necessarily a better 
school. Higher scores may be explained by 'out of school' factors such as the fact that students enter 
school with higher academic abilities . In other words, a school’s 'effectiveness' should be judged by 
its contribution to a students achievement independent of the student’s home background. In this 
sense, it is the ‘value added’ by the school to the student’s literacy, academic and social skills (Grisay 
and Mahlck 1991) which should determine its standing 

Research Paradigms: School Effectiveness , School Improvement and Improving Educational Quality 

From the discussion above it is clear that there are potentially many different approaches to the study 
of educational quality. According to Grisay and Mahlck 1991, three principal research paradigms can 
be discerned from the literature. These are: 

i. Experimental studies which measure the effect of an innovation, such as the introduction of a new 
curriculum or new teaching method, on the educational system. Included in this research 
paradigm are overviews, or meta-analyses, which re-analyse the size of the effects of a specific 
factor, for example the use of textbooks, on educational outcomes. 

ii. Large Scale, ‘input-output’ research, which sets achievement against natural variations in input. 
Non-school features of the input (age, IQ, home background, etc.,) are statistically controlled in 
order to identify the educational variables associated with better achievement. 



iii. Qualitative studies on school improvement, the so-called second generation school effectiveness 
studies (Riddell 1996) which rely on techniques of observation and case studies. These studies 
differ from first generation school effectiveness research in that they place less emphasis on the 
identification of input variables that can be altered separately (supplying more textbooks, 
increasing the numbers of teachers, improving their training) and focus rather on process variables 
and systemic factors (school climate, nature of leadership, style of management, teaching practices 
etc.). 

In developing countries, there have traditionally been far fewer examples of experimental research than 
in more industrialised countries. However, there are increasing examples of quasi experimental studies, 
mainly those carried out as rigorous evaluations of substantial educational innovations. Currently, the 
World Bank is carrying out a number of such evaluation studies in developing countries based on the 
use of randomised control groups (Newman et al, 1997). 

The most common research tradition in developing countries is still that of the input-output survey. 
However, there is increasing criticism of the value of these studies in determining the quality of 
education in developing countries. 

Jansen (1995) has reviewed the field of School Effectiveness research in developing countries and 
concluded that it does not address the central question necessary for the development of educational 
quality in the developing world. He argued that school effectiveness research has reached a 'cul-de-sac 
in the 1990s'. 

Jansen is concerned by the fact that most School Effectiveness research in developing countries 
continues to suffer from the criticisms levelled against 'first generation' studies of this kind, ranging 
from methodological critiques (Purkey & Smith, 1983) to 'insensitive cultural transfer' (Fuller & 
Clarke, 1993). 

The main methodological criticisms include, sample bias, narrowly focused outcome measures, lack of 
control for background characteristics, single level analysis of effects and so on. The trans-national 
character of school effectiveness research has also attracted a lot of criticism ‘insensitive transferral of 
methodologies from industrialised countries to developing country contexts. 

While Fuller and Clarke (1993) argue for a more 'culturally situated model of school effectiveness' 

(p.l 19), Jansen points to the growing dissatisfaction with the effectiveness paradigm and calls instead 
for a research paradigm which focuses on quality rather than effectiveness. He suggests that 'studies 
of effectiveness and studies of quality represent competing and incompatible agendas for school and 
classroom-based research' and makes a distinction between a 'school effectiveness' and a 'school 
quality’ approach. He suggests that education quality is 'concerned with processes of teaching , 
learning, testing, managing and resourcing which must be investigated on their own terms', i.e., 
through in-depth qualitative investigations of such processes and draw on more insider perspectives of 
what happens inside schools and classrooms. Thus, unlike Fuller and Clarke who argue that school 
effectiveness research should be broadened and 'culturally situated', Jansen dismisses the approach as 
being incompatible with assessing quality. He captures the differences between the two approaches as 
follows 



Figure 1 - Differences between school effectiveness and school quality approaches 
School Effectiveness School Quality 



Origins in economics, using the production 
function model 

Studies the effects of a set of inputs (e.g. 
textbooks) on a specified output (e.g. student 
achievement) 

Utilises large-scale statistical methods, e.g. 
multiple regression models to 'determine' the 
relative effects of different inputs on achievement 
Results are often aggregated for a large number of 
schools offering generalisations across contexts 



Influenced in part by anthropology, descriptive 
procedures 

Studies school and classroom level processes and 
their interactions, and the impact on achievement 

Uses ethnographic instruments adapted for 
particular contexts e.g., interviews, observation 
schedules, questionnaires etc. 

Results often specific for particular schools or 
classrooms, though generalisations are also 
sought across schools and classrooms 

From Jansen 1995:194 



Riddell (1996) who also reviews the value of school effectiveness research in developing countries is 
however much more optimistic about the future of the research paradigm in this context. She argues 
that the promise of a 'third wave' of school effectiveness research in developing countries is in danger 
of being lost without being fully understood or explored. Riddell points to 'a trickle' of research which 
has utilised multilevel analysis as a method, but adds that these studies have never really taken root. 

Riddell feels strongly that school effectiveness research has not been given an adequate chance in 
developing countries and that it is in danger of dying an 'untimely death'. Most critics, Riddell 
argues, are calling for a return to 'second wave' classroom-based observational studies. 

To this end she is critical of the stance adopted by Jansen in relation to research methodologies. She 
accuses him of playing quantitative and qualitative methodologies against each other and questions his 
belief that qualitative approaches are necessarily better for policy interventions than quantitative 
approaches (see Johnson, 1995). 

Riddell argues that the so-called 'second wave’ of school effectiveness research, which places emphasis 
on process variables, has been overlooked in developing countries although it has had marked success 
in industrialised countries. As such, she proposes that the third wave of school effectiveness research 
(multi-level modelling) offers a way in which the false dichotomy between quantitative and qualitative 
approaches can be bridged and that 'multi-level analysis is capable of doing what Jansen suggests we 
should be moving towards, i.e., the possibility of delineating complicated networks of cross-level 
relationships within classes, within schools and contextualising the different multi-cultural 
backgrounds in evidence'. 

The separation of qualitative and quantitative approaches in studying quality is a problem. Indeed, any 
research which sets out to fly a particular methodological flag regardless of the specific research 
questions being addressed is missing the point. The search for suitable methodologies to assess 
quality is an important consideration of the present study. It is thus useful to review a small selection 
of studies on educational quality in developing countries , which span the continuum from quantitative 
input-output studies to more qualitative, case study approaches. 

Case studies of school effectiveness, school improvement and educational quality 

1. IIEP Primary School Quality Study 

This national study of primary education in Zimbabwe was carried out in 1980. The aim of the 
research was to produce a list of the most effective schools in Zimbabwe and target these for further 
study. The study was conducted by the International Institute of Education Planning (IIEP) as part of 
an in-country training exercise in the development of research skills (Ross & Postlethwaite, 1992). 
According to the report, the study was concerned with the level and distribution of inputs to schools 
which were considered by the Ministry of Education and Culture to be central to the provision of basic 
education. Thus data were collected on school buildings, teachers and their living conditions, and 
resources in schools and classrooms. The study was also interested in measuring student outcomes and 
a reading test was designed as an outcome variable. 



The sampling strategy ensured a representation of urban and rural schools, government and non- 
government schools stratified by region. One hundred and fifty schools were selected on this basis. To 
administer the reading test, 20 pupils from grade 6 classes were chosen randomly from each of the 150 
schools. 

Schools were ranked according to average reading attainment scores as well as the average socio- 
economic level of the school (these indicators were derived from a questionnaire completed by the 6th 
grade students and based on seven possessions at home). After controlling for student economic 
status, school averages of the residual scores were computed in order to rank them from 'the most 
effective to the least effective’. 

Many criticisms of the IIEP study have been made, most notably from Riddell (1997) who points out 
that the study ignores several elements of good research design for school effectiveness studies . 

Perhaps the two most important omissions of the IIEP study are, first, the failure to employ a multi- 
level analysis of the data rather than the single level analysis performed, and second, as argued by 
Riddell, the problem of constructing valid and reliable baseline measures of student level intake. 

2. Namibian National Learner Baseline Assessment 

The Namibian National Baseline Assessment was conducted in 1992 to answer the question ‘how 
much do Namibian children leam in schools?’ (Namibian Ministry of Education and Culture, 1994). 
The purpose was to collect information about the English and Mathematics proficiencies of students. A 
nationally representative sample of 136 primary schools was selected. One grade 4 class and one Grade 
7 class were randomly selected from the 136 schools and all the pupils in these classes were given 
criterion-referenced tests in English and Mathematics. The only other information gathered was the 
age, sex and home language of the pupils surveyed. 

The baseline study provided information of student learning beyond that which teachers already had 
provided information about in schools which allowed their comparison to others in English and 
Mathematics proficiencies. The study also suffered methodological problems including a lack of 
school-level intake measures. 

The two studies described above, according to Riddell (1996), fit the model of school effectiveness 
study, which Cuttance (cited in Riddell, 1996) describe as the 'Standards Model’. The Standards 
Model is simply a league table which compares the average performance of pupils in a given school 
with the average performance of pupils across all schools. By way of contrast, the second and third 
models outline by Cuttance, the so-called School-level Intake Adjusted Models' and 'Pupil-level 
Intake Adjusted Models’ could have been achieved in the IIEP study if school achievement (derived 
from the reading test scores) had been subjected to a regression analysis in relation to the school-level 
socio-economic data. The residuals would have constituted a ‘school-level intake adjusted measure’ (see 
Riddell 1996:12). 

3. The Botswana Junior School Study 

The Botswana Junior school study was carried out by Fuller, Hua and Snyder. It was aimed primarily 
at establishing the extent to which the achievement of girls is affected by particular classroom practice. 
This study represents a combination of qualitative methodologies based on extensive classroom 
observations of 2 1 4 teachers in 31 junior-secondary schools and outcome measures derived from 
achievement tests in English and Mathematics administered to the same students on two occasions, the 
second occasion occurring a year after the first. To its advantage, the study has a pupil level intake 
measure. Four sets of variables were used: material conditions and classroom inputs; teacher 
characteristics and training; teachers pedagogical beliefs and efficacy; and teaching practices and 
classroom rules. According to Riddell (1996), the study represents one of a small number of 
sophisticated school effectiveness designs in the third world. 

Although, the results of the study are disappointing - it would appear that teachers asking 'open-ended 
questions' reduced the gains in maths scores for girls - the study did however raise important questions 
about the cultural relevance of certain pedagogical practices deemed to be 'effective’ in the context of 
industrialised countries. 



4. World Bank Project Designs and the Quality of Education in Sub-Saharan Africa 

The study carried out by Heneveld and Craig (1996) abandoned quantitative approaches to school 
effectiveness research for a qualitative approach to monitoring effective schools. Heneveld and Craig 
took as given the different sets of factors related to school effectiveness which had arisen in various 
studies, including, Dalin (1994), Fuller (1987), and Schreens and Creemers (1989). Based on these 
factors, the authors developed a conceptual framework based on 18 key factors that influence student 
outcomes. The factors were divided into four categories - supporting inputs from outside the school 
and enabling conditions, i.e., school climate and the teaching and learning process inside the school. 

A sample of World Bank projects was selected on the basis of the degree of attention paid to the 
characteristics of school effectiveness identified in this conceptual framework. Twenty-six country 
projects were eventually identified and appraised in this respect. 

Riddell (1996) remarks that the Heneveld and Craig study is different to that of the Botswana Junior 
School Study which queried, rather than accepted, the relevance of school effectiveness factors in 
developing countries. She also remarks that the study does not look directly at student achievement 
but concedes that Heneveld and Craig stated that they had no intention of doing so. Rather the 
intention of their framework seeks to assess the presence and dynamics of conditions that have been 
identified as conducive to effective education. Riddell (1997) is adamant however, that the conceptual 
framework leaves itself open to misuse by those 'unschooled in its dangers’ and argues that ranking 
schools using Heneveld’s descriptors to monitor educational quality based on ‘conducive conditions’ is 
entirely different from ranking effective schools based on the relationship of such conditions to pupil 
outcome (1997: 195). 

5. How schools Improve 

Dalin’s (1994) study of 'How schools improve’ is a qualitative study carried out in Bangladesh, 
Columbia and Ethiopia. In-depth studies were made of 31 rural primary schools. On the basis of the 
following indicators: 'degree of implementation of key aspects of the reform; degree of impact on the 
students, teachers and the school as an organisation; and the degree of institutionalisation of the reform 
or the 'routinisation' of practices'. Schools were sorted into three categories (excellent, very good and 
good schools). 

The study sets out to account for educational change in a much broader sense than student outcomes in 
the quantitative research tradition. Also, the study gave credence to a wide range of perspectives of 
change. For example it contrasted the perspectives of key informants with those of local informants. 

The study is an important addition to a relatively limited number of qualitative studies of educational 
quality. The focus is on accounting for educational change and its strength is the importance given to 
different perspectives on the change process. 

6. Profiles of Learning in South Africa 

A study carried out by Johnson (1998) sought to develop a framework in which education quality 
could be assessed through the use of 'insider perspectives’ (Little, 1995). Working with South African 
primary school teachers, the researcher developed a set of indicators of achievement for literacy and 
numeracy within five broad levels of progression: initial, developing, independent, complex and 
advanced. These levels are meant to be reflective of increasing competence and when they are related to 
children’s age, the resultant matrix essentially constitutes a 'growth model’ (Rowe and Hill, 1996) 
which projects a child’s developing levels of performance. 

An illustration of the levels of achievement is provided in figure 2 below. According to the matrix, the 
average seven year old child for example, could be considered to be working within the 'independent’ 
level. The 'O’ indicates the average or median while the 'whiskers’ indicate the range. Thus some 
seven year old children may still be achieving within the 'developing’ level while other may be 
progressing towards the 'complex’ level. 
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Figure 2 - The Profiles of Learning Framework 



Levels 




3 4 5 6 7 8 9 10 11 12 

Age in years 



From Johnson , 1988: 388 



This approach to assessing educational quality through a developmental or growth framework is 
summed up by Masters (1994) as one which: 

“seeks to provide a more explicit identification of outcomes and a framework against which . . . 
the progress of an individual, a school, or an entire education system can be mapped and 
followed. But this approach is built not around a notion of an outcomes check list, but around 
the concept of growth ... Student progress is conceptualised and measured on a growth 
continuum, not, as the achievement of another outcome, on a check list” (Masters, 1994: 9, cited 
in Rowe and Hill, 1996: 315). 

Johnson (1998) shows how teachers, on the basis of evidence assembled through portfolios, profiled 
the learning achievements of their students. The difficulty with the study is that it was not able to 
calculate the achievements of all learners. Rather, it demonstrated, through case studies, what selected 
children were able to achieve. These data are thus indicators of learning achievement of children in 
different age categories, but not overall scores. 

The six studies reviewed above are examples of approaches to assessing educational quality from a 
range of methodological perspectives. The IIEP study in Zimbabwe, and the Botswana Junior School 
study are both examples of input-output research. Student achievement is of concern only in relation 
to the inputs provided. Heneveld and Craig's study and that of Dalin both adopt qualitative 
methodologies. In the case of Heneveld and Craig, they review World Bank projects in relation to 
effectiveness indicators derived from the existing School Effectiveness Literature. Dalin adopts a case 
study approach and is interested in processes. Johnson's study in South Africa is an example of a 
study which draws more substantially on insider perspectives. Although limited in its methodology to 
provide overall scores of achievement, it is distinguished by being part of a tradition of developmental 
and interventionist research which seeks to set up activities designed not only to understand the 
underpinning factors which promote quality in an education system but also to enhance it. 

It is on this latter, more interventionist approach, that the present study is based. A central concern is 
the development of a profiling system in which student progress can be monitored. However, 
underpinning this aspiration to develop more meaningful and manageable systems for recording student 
progress, is the increasingly powerful message now emerging from the literature concerning the 
capacity of assessment to be, in itself, a mechanism for enhancing quality. As Black and William 
(1998) have shown, the sensitive use of formative assessment in the classroom can lead to a significant 
rise in the overall level of achievement of the class. The effective communication of learning goals to 
students, regular monitoring of learning progress and feedback to students that helps them understand 
how to improve, are key elements on this respect. Thus an exploration of teacher’s current practice in 
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this respect in the countries studied, and how good practice could be developed and encouraged, was 
another central concern of the Assessing the Quality of Learning and Teaching in Developing Countries 
Project. 

This objective was felt to be particularly important in an international context in which concern with 
the measurement of quality, not nationally and comparatively, seems in danger of eclipsing the 
arguably more important questions concerning how such quality can best be promoted at individual 
pupil and classroom level and hence ultimately the school and system level. 

Another important feature of this study, therefore, was the aspiration to combine the insights into how 
best to both monitor and promote quality which the school effectiveness research tradition has 
generated, with those which have arisen from the more specific focus of assessment research concerning 
how it can best be used to support learning (Gipps, 1994). 
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Chapter 3 - Country Profiles 



The research was carried out between 1996 and 1998. Two countries, Malawi and Sri Lanka were 
selected to participate in the study. The principle was to include one country in Africa and another in 
Asia in order to provide widely varying cultural contexts. Moreover, there are notable differences 
between the educational systems of these two countries, not least that Sri Lanka offers a reasonably 
established educational system while in Malawi, there has been considerable effort to 'renew’ the 
education system. 



Malawi is a narrow, landlocked country in Southern Africa, bordered by Mozambique, Zambia and 
Tanzania. The country is divided into 3 administrative regions - North, Central and Southern - that 
reflect historical, socio-cultural and political differences. The country gained independence from 
British colonial rule in 1964, and the first multi-party elections held in 1994 saw an end to the one- 
party regime of Dr H K Banda. 

Moleni (1999) conducted a study in Malawi and found that urbanisation is increasing rapidly, although 
over 85% of the population is still found in rural areas. The southern region consisting of Mulange, 
Blantyre and Zomba is the most densely populated (UNIMA/SARDC 1997) The main urban centres 
in the South are the industrial capital of Blantyre-Limbe and the university town of Zomba. 
Infrastructure is poor, with few tarmac roads and limited access to public transport. Less than 1% of 
the rural population has electricity and access to clean water is problematic for both rural communities 
and urban townships. 

The history of education in Malawi is interesting, Grant-Lewis et al (1990) claim that formal 
schooling was established by Christian missionaries based on their ambition to evangelise. In this 
respect, education for women was deemed to be unnecessary, since women could not become preachers 
(Grant-Lewis et al, 1990). The colonial government reinforced this and during much of Banda's 
regime, female participation in education remained severely restricted. This is reflected in an adult 
literacy rate today of less than 42% for women. (DFID 1998) 

In 1994, following the democratic elections, the new ruling party, the United Democratic Front (UDF) 
stressed its commitment to education, regarding it as crucial to their central policy of Poverty 
Alleviation (MoE 1995). The new government quickly put in place a number of sector-wide 
educational reforms, most notably, the abolition of tuition fees and unifonns at primary schools. 
Expansion of secondary schooling rested on a programme to build mixed, day secondary schools and 
closer partnership with the private sector (Swainson et. al. 1998). 

Tizora (1999) (cited in Moleni, 1999) claims that enrolment rose from 2 million to over 3 million 
following the introduction of free primary education in 1994, with a net primary enrolment of over 
85%. Significantly, gender disparities in enrolments narrowed during the 1990's (Swainson 1998). In 
1995/96 girls comprised 49% of new admissions in standard 1, although this percentage reduces as 
girls get older by the time they reach Standard 8, they make up 39% of the enrolment (MoE 1996). 
However, high net enrolment rates mask poor internal efficiency due to late entry, high drop-out rates 
and repetition for both sexes (Swainson et al 1998). 

According to Moleni (1989), transition rates to secondary education have increased in recent years, but 
remain low at 9.3%. Following completion of the Primary School Leaving Certificate (PSLC), 
students may be selected, on the basis of their PSLC results, to subsidised government or grant-aided 
secondary schools. Grant-aided schools are generally well established boarding schools run by Church 
organisations and are often single-sex institutions. 

For all intents and purposes, the education system in Malawi has undergone a transition, mainly in 
terms of quantitative expansion, but there appears to be a number of problems in relation to the 
improvement of educational quality. 

According to Weber and Chibwana (1999), increased enrolment had resulted in higher pupil-teacher 
ratios - currently 70: 1. Although the aim is to reduce this ratio in the long term, in the short term the 
government has adopted three measures. The first of these is a national teacher training effort aimed at 
training 20,000 new teachers. Second the conventional two-year residential teacher training programme 
has been replaced by a three-year programme in which trainees spend three months at the teachers’ 
college and the rest of the time in school-based, supervised teaching placements. Third, the school day 



Malawi 




ERIC 




9 ,., 



is being altered to accommodate ‘extended shifts’ which allow for more and, presumably, smaller 
classes. 

Large class sizes, resource shortages, and an inadequately trained teaching corps are seem to be 
associated with low achievement and wide-scale dissatisfaction with schooling (Johnson et al, 1992) 
and concomitantly low school completion rates. According to Weber and Chibwana (1999), only 23% 
of any given cohort completes eight years of primary school. Further, because of high repetition rates, 
it takes an average of 12 years to complete the eight-year primary school cycle (Ministry of Education 
and Culture, cited in Weber & Chibwana, 1999). 

A number of innovative projects have been established to raise the quality of education, but have been 
mainly on inputs to education. The study reported here is an attempt to develop school and classroom 
based procedures to raise the quality of learning, teaching and assessment. 



Sri Lanka 



By contrast to Malawi, Sri Lankan achievements in education are quantitatively impressive. The GNP 
per capital in Sri Lanka is US$753 (World Bank, 1996), but its social indicators are typical of 
countries with much higher incomes. According to a country paper produced by DFID (1997), Sri 
Lanka has a literacy rate of over 90% and the highest basic and secondary education participation in 
South Asia. Over the past 50 years, the expansion in school enrolment has been remarkable. 
Participation in grades 1-5 is now close to universal, while the proportion of pupils still at school in 
year 1 1 is close to 70%. Girls have participated in this expansion to an even greater extent than boys 
and gender ratios range between 92 and 95 in grades 1-5. In higher grades, the gender ratio seems to 
climb steadily so that by year 1 1, when students sit the O level examination, there are as many as 1 15 
girls for every 100 boys. 

The increasing public demand for education has resulted in a network of accessible schools spread 
throughout the island. Pupils have access to a primary school within a radius of 2 km, a junior- 
secondary school within every 5 km and a senior-secondary school within a radius of 7-8 km (Hart & 
Yahampath, 1999). 

According to the 1997 census, there are 10,120 schools in Sri Lanka comprising of four different types 
(see table 1). The student population is 124,108 with a total teacher population of 179,589. 



1AB 


1C 


Type 2 


Type 3 


Total 


515 


1,777 


3,877 


3,951 


10,120 



Type 1AB 
Type 1C 
Type 2 
Type 3 



Schools with Science classes 
Schools with Arts/Commerce classes 
Schools up to grade 1 1 
Schools up to grade 5 



Schools are divided into Sinhala and Tamil medium, with the latter accounting for 3% of the total 
number of schools (see table 2 below). 



Medium 


1AB 


1C 


Type 2 


Type 3 


Total 


Sinhala 


828,206 


1,097,280 


915,716 


314,705 


3,155,907 


Tamil 


212,832 


262,776 


286,833 


233,760 


96,801 



According to DFID (1997), the quality of education, particularly at the primary level appears to be 
declining. This is thought to be largely due to the recruitment from 1989 to 1995 of about 50,000 
mostly untrained teachers. A number of researchers, including Karyawasam (1991), Nanayakkara (1992 
and 1994) and Wijesuria (1994) (all cited in DFID, 1997) have carried out studies of primary school 
quality in Sri Lanka in recent years, and all have reported disturbing trends in levels of both maths and 
language achievement (DFID, 1997). Karyawasam, for example concludes: 

“The actual performance is not up to standard even in the best schools in both mathematics and 
language in all the districts ... The performance of type 1C, 2 and 3 schools is shockingly poor, so as 
to warrant immediate remedial action (1991: 27) (cited in DFID, 1997). 

There have been several innovations over the years to improve education quality (Little, 1991), starting 
with the introduction of an integrated curriculum in primary schools in 1972. Recently the 
government established a National Education Commission to study education standards and a 
Presidential Task Team has been formed to implement its recommendations. 
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Chapter 4 - Research Methodology 



This chapter describes the process of implementing the research project. The following issues are 
discussed. 

i. Setting up the research project 

ii. The operational framework 

Setting-up the research project 

Establishing the project in Malawi and Sri Lanka 

in Malawi, it was agreed that the Planning Section of the Ministry of Education be responsible for co- 
ordinating the Project and that the Centre for Educational Research and Training, based at Chancellor’s 
College, Zomba should be responsible for providing local counter-part research staff to support the 
technical aspects of the project. 

In Sri Lanka, it was thought appropriate for the research project to be based at the National Institute of 
Education. Counterpart research staff were provided by the Institute and given that it was necessary to 
work in three languages — English, Sinhala and Tamil, these counterpart researchers played a vital role 
in the technical aspects of the project with the teachers involved in the study. 

Selection of the localities for the study 

Decisions about selecting the locality for the study in each country were left largely to the local 
research partners but were guided by the following considerations. 

That the district / area be roughly representative of the school population on a continuum 
representing on the one hand, highly under-privileged rural areas to well developed urban areas on 
the other (G ovinda & Varghese, 1993: 19). 

In Malawi, discussions were held with selected Regional and District Education Officers and a number 
of districts was visited. It was decided that the Lilongwe urban and rural districts should be selected as 
these would provide a sample of schools which was representative of all types of schools in Malawi. 

In Sri Lanka it was important to ensure that the geographical area chosen for study was broadly 
representative of an ethnically heterogeneous population. 

The Kurunegala District in the North Western Province was proposed as that most representative of 
schools in Sri Lanka. Another important consideration was its accessibility and the extent to which 
local education authorities would be willing to co-operate. The Provincial Director of Education in the 
North-western Province was active in selecting the schools and later became a central figure in the 
research process. A number of schools in the district were visited. 



Selection of schools for the study 

As with the selection of the localities for the study, the selection of schools was left to the local 
research partners. The following guide was established for selecting schools. 

That the schools represented a reasonable variety in terms of size , and had varying levels of human 
and material resources and other conditions characterising the context of primary education in the 
country (Govinda & Varghese, 1993: 22). 

Ten schools were selected in each country. In Malawi five schools were selected from the Lilongwe 
Urban District and five schools from the Lilongwe Rural District. In Sri Lanka, of the ten schools 
selected, these were broadly representative of the types of schools (1 AB, 1C, 2 & 3) and included two 
Tamil schools. 

Selection of teachers f 

Thirty teachers were selected to participate in the study in each country. Because of the differences in 
the 'cycle’ of education and the organisation of schools in Malawi and Sri Lanka, the method of 
selection for teachers differed. 



In Malawi, from the ten schools selected to participate in the study, teachers were chosen to represent 
each of the three sections of the primary school - the Infant section (Standard 1 and Standard 2), the 
Junior Primary section (Standards 3, 4 and 5) and the Senior Primary section (Standards 6, 7 and 8 

In Sri Lanka, the pattern of school organisation is 5 years of primary education, 3 years of lower 
secondary, 3 years of secondary and 2 years of upper secondary. There is no division between primary 
and secondary schools and the majority of schools have both primary and secondary grades. Further, 
English is introduced as a subject for the first time in Year 3. 

It was decided to select only year 4 teachers. Thus, in each of the ten participating schools, two year 
four classroom teachers were selected. Further, the English teacher in each of the ten schools was 
selected. 

The operational framework 

The research was organised around the stages described in the operational framework outlined in chapter 
1. To recap, the research was organised in five broad phases. These are: 

i. Reviewing the learning requirements as stated in curricula, syllabi, and textbooks in each country 

ii. Establishing standards and norms based on the professional experiences of teachers and an 
emerging literature on learning 

iii. Developing procedures for assessment 

iv. Administering assessment tasks, collecting evidence of children’s achievements and recording 
outcomes 

v. Developing profiles of children's achievements 
These phases are described in more detail here. 

7. Reviewing learning and teaching 

The first phase of the research was to review the processes of teaching and learning in Malawi and Sri 
Lanka and principally, the expectations embedded in both systems as to what constituted 'successful 
learning'. In other words, we needed to understand what the requirements of Primary Schooling were - 
and what learners were expected to achieve at different stages. Thus the Primary School Teaching 
Syllabuses for English, Chichewa and Mathematics for Standards 1 to 8 were reviewed in Malawi. 
These syllabuses were examined in terms of their subject objectives. In addition textbooks in use in 
Malawi primary schools in English Chichewa and Mathematics were collected and the range of 
activities examined. 

Similarly, the syllabi for Year 4 in English and Mathematics were reviewed in Sri Lanka. At the time 
of the study, Sri Lanka had begun work on developing an Essential Learning Continuum (a criterion- 
referenced framework for the assessment of learning). 

From these documents, an impression was gained of the demands the curricula made on students. The 
question that arose for both countries was whether the demands of the curricula were unrealistic in their 
expectations of student’s achievements. This assumption needed to be tested. 

The processes of teaching were also scrutinised carefully. The teachers selected for the study were 
treated as focus groups and discussions about their professional practice offered useful insights into 
how learning was perceived and supported. A selected number of classroom observations were also 
carried out in particular to review the context in which Literacy and Mathematics were taught and 
learned. 

In sum, the findings from these reviews were: 

i. expectations of learning embedded in curriculum material, syllabi and textbooks were in the 
words of teachers 'ambitious', 'unrealistic', and 'unclear'. 

ii. The measures (tests and examinations) used to assess learning outcomes were 'too narrow' and 
'incapable of determining what learners know or can do'. 

iii. Approaches to supporting leaming(teaching) were limited, mainly due to the lack of resources and 
the quality of teacher training and dated classroom methodologies. 




2. Establishing standards and norms and developing assessment tasks 

The second phase in each country was designed to work with teachers with a view to developing a 
framework to profile learning achievement. Teachers were asked to concentrate on a limited number of 
Learning areas. These were confined to Literacy (speaking, reading and writing) and Mathematics 
(number, measurement, area, investigations). 

Meetings were held in both countries to review 'indicators of achievement' in Literacy and Mathematics 
derived from the assessment systems of a wide range of countries. Teachers were asked to develop, 
from their own professional understandings of what learners are capable of achieving, a hypothetical 
framework consisting of four levels on which children's learning could be profiled. Examples of how 
this was achieved in other countries were shared and discussed. 



In both countries, teachers developed a rudimentary framework consisting of the following levels. 



Level 1 
Level 2 
Level 3 
Level 4 



Basic Level 
Emerging 
Established 
Advanced 



Basic knowledge, skills, competencies and attitudes 
Developing or emerging skills and competencies 
Skills and competencies firmly established 
Advanced skills, knowledge 



Work was also begun, in a very rudimentary way, on developing procedures to gather evidence of 'what 
children could do'. This included thinking about ways in which teachers could use assessments in 
classes which were often as large as 120. The principle, was to involve teachers in the development of 
procedures for assessing students. 

5. Developing procedures for assessment 



Teachers began to develop an initial set of assessment tasks. It was necessary, however, for these tasks 
to be refined and developed further. The initial set of Assessment Tasks produced by teachers at an in- 
country workshop were brought back to the University of Bristol for further development. (The 
completed set of tasks is contained in a separate volume). It suffices here to describe briefly the 
purposes and nature of the assessment tasks: 

In Malawijhe researchers worked with groups of teachers from the Infant, Junior Primary and Senior 
Primary sectors to develop tasks in various areas of the English and Mathematics curriculum. For the 
first trial activity in schools, the researchers developed tasks in the number curriculum at the three 
levels. 



In Mathematics the purpose of the tasks was to provide data on what pupils knew and were able to do, 
on the difficulties they experienced by pupils and the mistakes which they made. To this end, three 
sets of tasks were developed at each level. First, there was a set of aural tasks where teachers would 
read questions to the class and the pupils would make individual responses on paper. Second, there 
was a set of tasks presented in written form for individual response by pupils. Finally, there was a 
group task which involved small groups of pupils working together and producing both written and 
oral reports on their work. Tasks were usually of a closed kind with a single correct answer the 
achievement of which required the pupil to possess specific from knowledge or skill, However, there 
were a number of less orthodox problems and more open questions which were intended to reveal pupil 
thinking. 

The Languages tasks were divided into reading and writing and included three tasks for each of the 
different levels. For the Infant section, Task 1 was an early spelling task designed to assess whether 
children could recognise and use single letters and groups of letters to represent whole words or parts of 
words. The second task assessed children's 5 capabilities as independent writers, while the third task 
assessed early reading. Here, the purpose was to find out whether the child understood that it is the 
print (the letters and words) which tell the reader what to read. 

For the Junior Primary section, three tasks were developed. Task 1 was designed to gain an 
understanding of children’s reading, particularly their independence, fluency, accuracy and 
comprehension. A reading record was created from a passage indigenous to the Malawian context and 
teachers were to use this record in gaining an impression of children as independent readers. Teachers 
were further required to make judgements about children’s comprehension by asking them to retell the 
content of the passage and by making predictions of what might happen next. The second task was 
designed to assess children’s capabilities as independent writers. This was divided into three types of 



task which were, an aural task where children listened to a passage read to them and they were asked to 
make up their own ending to the story. Task two required children to write a short story from a picture 
provided and Task 3 required the children to give a factual account which involved a description of 
their school. 

The Senior Primary Section were provided with reading and writing tasks. The reading task required 
pupils to interpret tables and diagrams in order to test their comprehension and the writing task 
demanded the production of a ‘personal account’. 

An accompanying set of Notes for Teachers was provided for each set of tasks. 

In Sri Lanka, the procedures involved working with the National Institute of Education (N1E), rather 
than with teachers directly because of the language problems. The NIE staff were organised 
appropriately to include three members each of the Language Committees in Sinhala, Tamil, English 
and Mathematics. As in Malawi, the task was to establish a rudimentary framework for profiling 
learning. 

The NIE staff engaged with the task of producing assessment tasks for Year 4 students. The 
orientation of the tasks was similar to those developed for Malawi. As was the case in Malawi, these 
tasks remained in fairly rudimentary form and were brought back to the University of Bristol for 
modification and further development. The tasks were to be returned to the NIE for translation into 
Sinhala and Tamil. 

Further meetings were held in both countries to finalise the tasks for implementation and to set up the 
basis from which the Centre for Educational Research (CERT) in Malawi and the National Institute of 
Education (NIE) in Sri Lanka could provide support to the teachers in administering the tasks. Both 
CERT and NIE organised meetings with teachers and trialed the tasks in the classrooms. 

Trials - Administering tasks, collecting evidence and refining the profiling system 

A major activity between the second and third phase of the study involved trialing the tasks. Teachers 
administered the assessment tasks in their classrooms to establish the usability of the tasks and 
children's responses to them. They also collected samples of children’s work. These samples were 
needed to refine the levels in the profiling framework. 

Once the first set of tasks had been administered, teachers in both countries brought samples of 
children's work to a meeting. In preparation for the meeting, teachers analysed the samples of work and 
awarded each piece of work a level. They wrote their justifications for awarding the different levels on 
the basis of answering questions about the piece of work. These questions were provided on the 
teacher record sheets for each task. 

At the meeting teachers worked in pairs and small groups and discussed the samples of work they had 
brought with them. They exchanged samples and re-rated pieces of work, all the time providing 
justifications, on the basis of the work in front of them, as to the levels being awarded. 

Through this process, teachers in each country and in each sub-group were eventually able to agree on 
samples of work which could be used as exemplars for each of the 4 levels in the profiling framework. 
These exemplars and other samples also provided teachers with the basis for developing statements to 
describe what might be expected at each level. These level descriptions were kept broad enough to 
accommodate most of the samples teachers had been exposed to over the course of the meetings. 

4. Administering tasks, collecting evidence and developing profiles 

The third stage of the research involved teachers and researchers in administering a second batch of 
tasks in their classrooms. This stage required the teachers to work independently to administer the 
tasks and to collect samples of children's work. 

Each sample of work was to be analysed by the individual teacher who was guided in doing so by 
questions included in the Teacher Record Sheets provided with each task. Teachers also had to write 
their own comments of the 'standard' the work had achieved and to award a level score on that basis. 



5. Profiling Learning 



The final stage of the research involved teachers meeting together in order to standardise the assessment 
framework and to ensure that their judgements were reliable and valid. 

Once again teachers selected samples of their pupils 9 work which they felt spanned all four levels. 

Once again, they worked in pairs and small groups where they exchanged samples and validated the 
levels awarded to each piece of work. 

When it was clear that the teachers’ assessments were standardised, i.e., that the reliability of their 
ratings was consistently high (inter-rater reliability was scored on Cohen’s Kappa) teachers were 
organised in groups to profile the entire sample of pupil scripts from all the sample schools. 
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Chapter 5 - Results 

Part 1 - Teaching and Assessment 

One aim of the study was to identify what kinds of teacher activities and what assessment practices 
existed in both countries. During the research period, a large number of observations was carried out 
and teachers were interviewed. 

The research was particularly concerned with finding out how literacy and mathematics were taught and 
assessed in the two countries. 

In Malawi the classroom activities observed showed that there was very little teaching of reading. 
Where reading activity was observed, teachers read to the whole class from text books. Listening to 
individual children read from fiction and non-fiction readers occurred in a small percentage of the 
lessons observed, although in most of the occasions observed, teachers were engaged in another 
activity, usually, marking exercise books, while children were called to the front to read. The rest of 
the class was asked to follow the text in their books. 

Only in a very few of the lessons observed did the teachers take a active role in reading. They tended 
to read 'with 5 children and helped them when they encountered difficult words. Even here though, the 
teacher was more concerned with correcting reading errors, rather than helping the child enjoy the book 
by drawing attention to characters, pictures, print and so on. 

No records were kept on children's abilities as readers. 

Teaching support for the development of writing was minimal. There was little evidence that teachers 
were engaging in activities designed to enhance and enrich the creative and factual writing abilities of 
their students. 

In only a small number of the lessons observed did teachers provide students with a variety of 'tools’ 
for writing. In some of the lessons observed, teachers provided resources for writing. This consisted 
of teachers 'recycling’ paper to make books for students to write in. This was particularly encouraging 
as in most cases, teachers pointed to the lack of resources for writing as a reason for limited writing 
activity. Even more encouraging was where teachers helped students to identify different audiences for 
their writing. In about a third of observed English lessons, teachers emphasised for example the 
differences between writing a 'friendly' letter and a 'formal' letter. Very few teachers viewed writing as a 
process’ where they encouraged students to produce more than one draft. This however, was not the 
norm in Malawi. 

The assessment of writing was focused on formal accuracy - grammar, spelling and punctuation - and a 
mark, usually out of 20, was awarded. Very few records were kept. 

In mathematics, teacher concentrated mainly on addition, subtraction, multiplication and number work 
of different kinds. Very few, if any, observations were made of children investigating or solving 
problems through group discussion. The assessment of mathematics consisted of marking answers 
right or wrong and very few records of children's progress were kept. 

In Sri Lanka it was clear that the context of learning and teaching was very different to that in Malawi. 
Schools were far better resourced and teachers were engaged in numerous activities to support both 
literacy and mathematics. 

Most encouraging to observe was established practice of recording student progress. Numerous 
innovative teacher made systems for recording children’s progress in literacy and mathematics were 
visible in all the research schools. One of the researchers on the team made the following remark: 

'In classrooms, I came across recording systems used by teachers to record pupil progress - 
achievement on several criteria judged at four levels, each half term. Progress over the year is 
monitored, in some cases, publicly displayed and, in one school, analysed and available in the 
Principal’s study. For one teacher, her record led to the short-listing of pupils who needed special 
help and these were monitored more closely with a more detailed set of criteria. In mathematics, 

I saw reports, or heard about, several more substantial surveys of what pupils can do at various 
stages in the primary school.’ 
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These findings are consistent with the view taken by Little (1991). According to her, continuous 
assessment was introduced as an alternative procedure to formal testing during the 1972 reforms in 
primary education, mainly to reduce the anxiety of sitting tests and to ensure that corrective measures 
could be taken quickly as soon as teachers realised the particular competencies each child failed to 
master. The use of assessment for formative, rather than summative, purposes was expected to help 
improve the competence level of pupils gradually. Teachers, however, need much training and 
commitment as records need to be kept carefully and regularly. 

Currently, continuous assessment is expected to be used in Years 1 - 3 while end-of-tenn formal 
testing is used in years 4 and 5. Although a format has been prepared, and guidance on filling it out 
has been provided, widespread use of these procedures is not evident (Little 1991). By contrast, end- 
of-tenn fonnal testing is evident in most schools, even for Year 1. 

It is clear from the above account that there is a considerable variety of assessment practices in use in 
schools in the two countries studied. This ranges from, at one extreme, teachers who are faced with 
such an overwhelming combination of huge class sizes and lack of even the basic resources that they 
are unable to employ any sort of systematic classroom assessment practice at all. At the other extreme 
are teachers whose professional skill and commitment has led them to develop their own procedures for 
maintaining pupils’ progress in order to support individual learning most effectively. 

Clearly, it would not be possible to devise a system that could meet this range of needs. However, 
these observations do underline the value of teachers routinely being provided with suitable assessment 
tools which would enable the majority to develop and operate much more useful systems for 
monitoring individual pupil achievement and for providing classroom-level feedback on the profile of 
student learning. 
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Part 2 - Assessing Literacy Learning 
General principles 

Literacy learning was assessed in Malawi and Sri Lanka. In both countries, both English and other 
national languages were assessed. In the case of Malawi, assessments were made of children's oral and 
written language and of reading in English and Chichewa. Similarly, in Sri Lanka, assessments were 
made of children's oral, written work and their reading in English, Sinhahla and Tamil. 

In each case, standard tasks, designed by the researchers and participating teachers were used to assess 
achievements. 

Tasks to assess literacy learning in Malawi and Sri Lanka 

The tasks were designed to assess children's abilities to communicate in different communicative 
contexts. In assessing Oracy and Reading, teachers were asked to select six to ten children in their 
classes on the basis of whom they thought were 'good', 'average' and 'weak' communicators. Once 
they had made their selection, they were to write the names of these children on Record Sheets which 
were provided. For the assessment of writing teachers were able to select all the children in their 
classes. Over the course of the research, the communicative competencies of the selected samples of 
children were assessed. Teachers recorded the results on the record sheets, and in the case of writing, 
collected corpora of texts from their students. 

Because of the diversity of the student samples, on these scores of children aged nine to 10 years 
(Junior primary in Malawi and year 4 in Sri Lanka are reported here). 

Developing a framework for the assessment of literacy learning 

As described in chapter 3, teachers met in groups on several occasions to discuss the samples of 
children's work. Each teacher brought to these meetings, samples of children's records (oral language 
was recorded verbatim on the 'teacher record sheet'), their own assessment of the level these samples 
had achieved and the teachers written comments of the work, justifying the level awarded. Teachers 
were encouraged to bring samples of those children's work who they had rated as 'doing well', 'average 
performance' and 'experiencing difficulties'. 

At the meetings teachers worked in small groups in which they exchanged the samples of student's 
work. The went about the task of studying these samples in order to determine what it is that children 
do when confronted by the given tasks. They looked at the comments written by the class teacher and 
the levels awarded to each sample. On this basis, teachers began to draft statements describing each 
level. 

Once the descriptions for each level had been decided, a second sample of records was selected. Each 
teacher was provided with a scoring sheet and rated each record. 

The level of agreement was high amongst teachers when these records were subjected to a statistical 
reliability procedure (Cohen's Kappa). 

This exercise allowed teachers to refine the level descriptions they had agreed on earlier to 
accommodate a wider variety of student work. 

The level descriptions differ between countries, but not markedly, and also between English and 
national languages within each country. The level descriptions for English in both cases were lower 
that the expectations expressed in formal curriculum documents of what students could achieve in the 
national languages at each level. 




The literacy scale below is an example for the assessments conducted in English in Malawi. 

The Literacy Scale and Level Descriptors 
Oracy 

Level 1 

The child shows awareness of the context and can name things in the illustration. The child is capable 
of making short sentences or statements about things in the illustration. 

Level 2 

The child shows an awareness of the context and can describe individual things or events in the 
illustration in simple terms. There is some evidence of the use of conjunctions. 

Level 3 

The child understands the context portrayed in the illustration and is able to describe events and items 
in the illustration in a simple and more complex sentences. The child is also begins to interpret the 
illustration and to reason or question beyond what is visible. 

Level 4 

The child understand the context portrayed in the illustration and is able to talk fluently about the 
events and items using a wide range of vocabulary and sentences in which meaning is embedded in 
complex ways. The child is able to interpret the illustration and reason and question what is not visible 

Administering Oral Language Tasks in Malawi 

The tasks comprised of an illustration, taken from language books for ten year olds children in both 
countries (see example on the next page). The selected students were to look at the illustration and talk 
about it. Teachers were provided with Record Sheets and recorded what children said verbatim, as far as 
was possible. 

An illustration of the task is shown overleaf, and the instructions to teachers and teacher record forms 
are contained in the appendices. 

The Samples 

In Malawi, 270 samples of children's use of English language were collected and the same children 
were assessed again in Chichewa. The age range was nine to ten year olds. In Sri Lanka, 60 samples 
of children’s oral use of English language were collected; 20 samples of children’s oral use of Tamil; 
and 80 samples of their use of Sinhala were collected. 

Interpreting the Literacy Profiles 

The following literacy profiles were constructed through an analysis of all samples of children's work 
in relation to the literacy levels developed in each country. 

The results are not intended to be compared between countries nor between languages. The profiles 
show however what children are achieving in each language, in each country, in relation to the expected 
standards set by teachers. 

Even though level statements are different for each country and within each country in relation to 
languages, teachers set an arbitrary standard, level 3 in each case, as the desired norm to be achieved by 
at least 50% of children. 

This gives us a basis for interpreting the results of each country in terms of children's achievements in 
relation to the benchmarks set for each learning area. 

Assessing Oracy: Malawi 

The illustration on the following page shows the task used to assess the use of o$al language in 
Malawi. The profiles of achievement are shown below. ^ 



Illustration of the Lanuage Task - Assessing Oracy in Malawi 



ENGLISH - JUNIOR PRIMARY 
ILLUSTRATION FOR ORAL TASKS 
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Profiles of Achievement 
Speaking in English and Chichewa 



Profile of Oral Competence - English 




Figure 1 shows that 22 percent of children achieved Level 1. 38.8 percent of children achieved Level 2 
while 27.7 percent achieved Level 3. Only 11.1 percent of children achieved Level 4. 



Profile of Oral Competence (Chichewa) 




Levels of Attainment 

Figure 2 shows that 12.2 percent of children achieved level 1. 32.9 percent of children achieved level 2 
while 40.7 percent achieved level 3. Only 14.4% of children achieved level 4. 



EXAMPLES OF INDIVIDUAL CHILDREN'S PERFORMANCES 
FOR EACH LEVEL 



The following examples drawn from oral samples in Malawi gives an illustration of the kinds of work 
considered to be at levels 1-4. In these examples, the verbatim records as written by the teachers are 
illustrated in the boxes below and the teachers comments, justifying their decisions to award a 
particular level, are also reproduced. 

Level 1 

Box 1 - Example of verbatim record of oral language 



K A bicycle, a pump, a tube, tyre , basin , water, bell, carrier, shoes , two boys , trousers , puncture . 

I am mending my bicycle. To find the puncture wika tavala m’madzi apez.e puncture ndi kumata , ndi 
kupopa ndi kukwera' . 



In this example, the child simply names items in the illustration. An attempt is made to construct the 
events illustrated in the picture but the child fails to follow through completely in English and 
completes the description in Chichewa. 

The teacher has this to say about the child's performance. 

Box 2 - Teacher comments and reasons for awarding level 

Chifuniro is weak . He only mentions parts of the bicycle and other things he sees in the picture. But 
he fails to describe what is going on in English 

He knows what is going on thus mending a puncture but he fails to put the description in proper 
language. 

His imagination is quite real according to what the illustration is about . 

The information given is relevant to the illustration . He knew that it was about mending a picture'. 
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Level 2 



Box 3 - Example of verbatim record of oral language 



I can see a man , a pump , a tube , a tyre , basin of water, handles , a chair, a band, a chain . 

They are mending my bicycle . tube has a puncture . Aika m'madzi kuti awone chibowo ndi f 
kumata . a kemmata aika mu tyre ndi kupopa \ 



In this example, the child names items in the illustration but there is an attempt to construct simple 
sentences. The child also is clearly able to construct the events illustrated in the picture but like the 
child above, fails to follow through completely in English and completes the description in Chichewa. 



The teacher has this to say about the child's performance. 

Box 4 - Teacher comments and reasons for awarding level 

' This pupil is not quite good or weak but average because he can understand what is taught and he does 
well in certain aspects. 

In this issue he only lacked vocabulary to expand his illustration . 

The information given was more relevant to the illustration 
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Level 3 



Box 5 - Example of verbatim record of oral language 



'/ can see a basin of water, a tyre , a tube , a pump, a boy , c/ man , spokes ; chain. 

The boys are mending a bicycle. A kupopa kuti apeze puncture, take the tube from the tyre then put in 
the basin of water to find where the puncture is . 

Then atenga mwale kupalira chibowa ndi kumata ndi mphira. Ndiye alowetsa mu tyre 
ndi kupopa 



In this example, there is evidence of an attempt to construct simple sentences. The child is also able to 
construct and interpret the events illustrated in the picture even where the actions aren’t themselves 
illustrated. For example the child say 'take the tube from the tyre then put it in the basin of water to 
find where the puncture is’. The child also lapses in to Chichewa to complete the description. 



The teacher has this to say about the child's performance. 

Box 6 - Teacher comments and reasons for awarding level 

'This girl is quite good. She tried to mention all what she sees in the picture. She was also able to say 
what was being done in the picture - only that she failed to say everything in English. The information 
given was quite relevant to the illustration. She said everything about mending a puncture and it was 
good'. 
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Level 4 



Box 7 - Example of verbatim record of oral language 



' Once there was a boy who was walking along the road, at the same time he saw a man sitting with his 
bicycle and the boy said . ' hello sir ; what are you doing?' 

7 am mending my bicycle ' the man answered. 

' Can I help you ' a boy asked. ' what should I do? 

First take out the tube from the tyre and find out the puncture. / will prepare it for you' 

So the boy is helping the man. 



In this example, the child is aware of the context of events. She is able to identify what is happening 
in the illustration and describes these in clear sentences. The child has expanded on the events and has 
brought her imagination into play. She talks of events 'before ' those illustrated in the picture, thus 
widening the context. 

The teacher has this to say about the child's performance. 

Box 8 - Teacher comments and reasons for awarding level 

'This girl has described the situation in an imaginative way. The description of the picture is true and 
relevant. She has expanded upon the illustration and her language is very good. 
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Assessing Oracy: Sri Lanka 

Tasks to assess oral competencies in Sri Lanka 

As in Malawi, the tasks were designed to assess children's abilities to communicate orally in different 
communicative contexts. Here too, teachers were asked to select six to ten children in their classes on 
the basis of whom they thought were 'good', 'average' and 'weak' communicators. Once they had made 
their selection, they were to write the names of these children on a Record Sheet which was provided. 
Over the course of the research, the communicative competencies of the selected sample of children 
were assessed. 



Administering The Task 

The task comprised of an illustration, taken from a reading book in for ten year old children in Sri 
Lanka - 'Let's Go to the Fair'. The illustration is of a mother and her two children buying fruit from a 
fruit stall. The selected students were to look at the illustration and talk about it. 

The English task is shown overleaf. The tasks in Tamil and Sinhalese are contained in the appendices. 

The Sample 

60 samples of children's oral competence in English were collected. 20 samples of work were 
collected from Tamil speaking schools and 80 samples of work from Sinhalese speaking children. 
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Profiles of Achievement 
Speaking in Tamil, Sinhala and English 



Profile of Oral Competence - Tamil 




Attainment Levels 

Figure 4 shows that 50% of students are achieving levels 1 and 2 in oracy in Tamil. 20% are 
achieving Level 3 and 30% level 4. 



Profile of Oral Competence - Sinhala 




Figure 5 shows that 26.4 % of children are achieving Level 1, 43.3% Level 2, 20.7% Level 3 and 
9.4% Level 4. 
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Profile of Oral Competence - English 




Figure 6 shows that 27.7% of children achieved Level 1, 42.5% Level 2, 20.3 percent Level 3 and 
9.2% Level 4. 



Profile of Oral Competence (Tamil, Sinhala and English) 




Figure 7 shows the profiles of achievement in oracy for all three Language groups. 
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EXAMPLES OF INDIVIDUAL CHILDREN’S PERFORMANCES 
FOR EACH LEVEL IN ENGLISH (SRI LANKA) 



The following examples drawn from oral samples in Sri Lanka gives an illustration of the kinds of 
work considered to be at levels 1-4. In these examples, the verbatim records as written by the teachers 
are illustrated in the boxes below and the teachers comments, justifying their decisions to award a 
particular level, are also reproduced. 



Level 1 

Box 9 - Example of verbatim record of oral language 

'There are house. There are bad. There are boy. There are two birds. Play cricket. Tree big. It is 
sun. There bad (bat). House. Sun sky 



Teachers were asked to make a judgement of each child's writing proficiency and to complete notes on 
each child according to the following questions: 

1. Does the child talk fluently? 

In this example, the child simply names items in the illustration. According to the teacher the child 
did not talk fluently, and hesitated very much. She also mispronounced many words. 

2. Does the child describe the illustration in an imaginative way (describing imaginative events) or 
simply describe the objects in the picture? 

According to the teacher for the example given above, the child showed little imagination and simply 
tried to name the objects in the picture. 

3. Is the information relevant to the illustration? 

In this example the child confined their description to what they could see in the illustration. 

The teacher completed the record as follows: 

Box 10 - Teacher comments and reasons for awarding level 

This child is very weak in building up sentences. Most of her sentences were grammatically wrong e.g. 
there are house, there are boy. She didn't have confidence at all. In order to make her speak I had to 
point to the objects in the picture. Therefore I judged this child to be at a level one in speaking. 
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Level 2 



Box 1 1 - Example of verbatim record of oral language 

This is a picture . There is a big tree . There is a sun . There is a house . There are birds in the sky. 
There are flowers . There are boys . They are playing cricket. They are not happy. There are windows. 
Tall boy is wearing a T-shirt. There is a roof. There is a fence. There are birds. There are mountains. 
There are trees. The sun is in the sky. There are clouds. 



T Does the child talk fluently? 

According to the teacher the child spoke without any hesitation and her speaking was fluent. 

2 . Does the child describe the illustration in an imaginative way (describing imaginative events) or 
simply describe the objects in the picture? 

According to the teacher the child described the illustration in an imaginative way e.g. 'there are two 
boys. The boys are playing cricket. The ball is in the window.' 

3. Is the information relevant to the illustration? 

In this example all the information was confined to what was visible in the illustration and as such 
relevant. 

The teacher completed this part of the record as follows: 

Box 12 - Teacher comments and reasons for awarding level 

In building up her sentences she made a good attempt. She gave meaningful sentences relevant to the 
picture but she made some grammatical mistakes in describing what has happened . Her description was 
good. Her pronunciation was good and her description of the picture was mostly accurate. On the basis 
of this evidence I judge her to be Level two. 
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Level 3 



Box 13 - Example of verbatim record of oral language 



'This is a garden. It is a sunny day. The time is twelve o'clock. There is a big tree . There is a ball. 
There is a wicket. There are two boys. There are flowers. There are birds. There are mountains. 

There are clouds. There is a fence around the house. The boys are playing cricket. They are not happy. 
The ball break the glass. The tall boy is wearing a T-shirt. One boy is wearing pads. 



1 . Does the child talk fluently? 

According to the teacher the child spoke very fluently and her pronunciation was good. 

2. Does the child describe the illustration in an imaginative way (describing imaginative events) or 
simply describe the objects in the picture? 

The child described the picture in an imaginative manner and the teacher points out that although the 
child described the situation in the picture she used her imagination on several occasions e.g. 'the boys 
are not happy'. 

3. Is the information relevant to the illustration? 

In this example all the information provide was relevant to the illustration. 

The teacher completed this part of the record as follows: 

Box 14 - Teacher comments and reasons for awarding level 

In building up her sentences she used the same structure e.g. there is..., there are... By looking at the 
position of the sun she guessed the time of day correctly. The child says ' There is a sun. The time is 
twelve o'clock.' She made one mistake i.e. 'the ball break the glass.' but overall her speech was vety 
good. This girl wanted to describe the situation not the objects. Her sentences are very meaningful. I 
think she is a good girl in her speaking therefore she was judged to be level 3. 
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Assessing reading in Malawi and Sri Lanka 



Reading Tasks 

The reading tasks were designed to gather evidence of children as independent readers. The task 
provides a broad assessment of children's reading, allowing them to demonstrate their ability to read 
aloud from a text, show what they have understood and give a personal response. 

Reading books in Malawi and Sri Lanka used at the levels of education for the children involved in the 
study were selected and extracts were selected to serve as the reading task. 

A 'running record’ was developed, (a matrix containing a copy of the text to be read , with each word 
contained in a separate box in the matrix and with blank spaces beneath each word). Teachers would 
follow the reading and mark the record for each reader, indicating accuracy, self correcting strategies, 
words which are omitted and so on. 

An assessment record sheet was also provided for each reader. On this form, teachers would record 
comments on how the child read, what they understood from the text and their attitude towards the text 
itself. 



The Literacy Scale and Level Descriptors (Reading) 

The following levels and level descriptions were decided upon by teachers in Malawi. A similar set of 
descriptions for each level was decided upon by teachers in Sri Lanka. It is worth noting again that 
there were slight differences in what teachers expected reading levels should be in English and for 
national languages. 

Level 1 

The child mis-reads most words and has no strategy for reading unfamiliar words. S/he does not show 
an awareness or an understanding of what s/he is reading. 

Level 2 

The child recognises some words consistently and shows awareness of punctuation. The reader shows 
an appreciation of the text. 

Level 3 

The child reads with accuracy and shows understanding of what he or she is reading through 
expression, volume and emphasis. She expresses opinions about the text. 

Level 4 

The child reads with accuracy, shows understanding by reading through expression. S/he has the 
ability to express opinions about main events and ideas. 
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Assessing Reading: Malawi 



The illustration on the following page is an example of a task used to assess the reading of 10 year 
olds in Malawi. 

Their achievements are profiled below. 
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An example of the reading task - Malawi 



ENGLISH - JUNIOR PRIMARY 
READING TASK 




Dziko and the msuku tree 




Dziko lived in a village near a big forest. He was a strong boy, but he did not like to 
work. 

One day he sat under a tree in the forest. Suddenly he heard a noise. He listened. 
Somebody was singing, but Dziko did not know the song. 

He stood up. Then he saw an old man. He was talking to a tree. He said, 'Mzuku tree, 
my children want something to eat. They are hungry. But I don't have flour or fish. 
Please, help my children. They want some nsima and fish. 
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Profiles of Achievement 
Reading in English and Chichewa 



Profile of Reading (English) 




Figure 8 shows that 13% of children achieved Level 1, 37.6, Level 2, 32.4, Level 3 and 17% achieved 
Level 4. 



Profile of Reading (Chichewa) 




Levels of Attainment 



Figure 9 shows that in Reading Chichewa, 6.5% of children achieved Level 1, 30.5, Level 2, 39.2%, 
Level 3 and 23.8% achieved level 4. 
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Profile of Reading (English and Chichewa) 
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Level 1 Level 2 Level 3 Level 4 

Levels of Attainment 



Figure 10 shows the profile of achievement for Reading in English and Chichewa. 
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EXAMPLES OF INDIVIDUAL CHILDREN'S PERFORMANCES 



The following examples of teacher comments based on evidence from both the running records and 
reading records of children’s reading in Malawi, provides an illustration of how decisions were made in 
awarding different levels. 

The example shown here is that of a child who was awarded Level 2. 
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ENGLISH - JUNIOR PRIMARY 
READING RECORD 



Child's name: Class: 

Class teacher's name: 

Name of school 



Dziko 


lived 


in 


a ' 


village 




















forest 


He:?%Sfe 


















'■'S- strong 


boy. 


. but 


he . '-".y: 


did 


: not" ; i:; 'V s 


1 










like,. 


work. 


.. One 


day : >XT ; 


he 


'1: sat 




A 










under .. 




tree 


in: x:. : 


the 


L forest "; . 


.. ! 










1 


Suddenly. 


he 


. heard 


a 


noise. 


. He 














i; listened. 


Somebody 


was 


singing. 


but: 


Dziko .v 














did 


not 


< know 


the 


song.. 


. He 




1 








stood 


up. | Then 


he . 


saw 


an 














old 


man. 


r He 


was 


talking 


to 




1 








a 


tree. 


He i said. 


'Msuku 


tree. 






i 


/'i 




my 


children 


want 


I somethin? 


to 


eat. 












They 


are 


hun^rv. j But 


I 


don't 






i 






have 


anv 


flour 


or 


fish. 


Please, 






A Floor- 








help 


my 


children. 


They 


want 


some 








1 


1 


nsima 


| and 


fish. 







I ! 



What to do 

1. Complete the reading record as follows: 

A For words that the child attempts but cannot read (when he or she makes an 
incorrect attempt at a word, write in the space what he or she actually said). 

X For any words that the child cannot read and does not attempt. (When a 
child makes an incorrect attempt at a word, write in the space what is 
actually said.) 

2. Comment on the child's reading performance. Use the set questions to help you. 
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ENGLISH - JUNIOR PRIMARY 
READING RECORD 



Analysis continued . . . 



a) Does the child respond to what he or she is reading through expression? 

b) Does the child read with accuracy? 

c) Does the child show an awareness of punctuation, for example by pausing 
in appropriate places? 

3. Ask the child to retell the story in his or her own words and to predict what might 
happen next. (Ask the child some questions like: which aspects of the story did 
you enjoy or not enjoy?) 

Make a judgement about whether the child is able to re-tell the context of the 
passage, including, at least, two of the main points and whether s/he can make a 
sensible prediction about what might happen next. 



Write your comments here: cJrJuLJ CU&L&_ 'Co /^L~Zg X jL 

hU 10 Xr£f>r , . 

APiD / r-ZjhjcXj&J ~U&Jt 

cUsxo OLruudSLT T~> r ^ Yy i 

Pa/vcu^e. /• ^ r^asJ UitC^ &<UUifasU( lluXL 

note-' to UM£XuclE^£w XU> l\JZ, lOQy) 



4. Make a judgement about children's abilities to talk in simple terms about the 
extent to which they find reading interesting or enjoyable. You could, for 
example, ask the child questions like: which aspects of the story did you 

enjoy /not enjoy? 



Write your comments here: 



q ~Ulc 



'1-UtL s 






EM 






C&nfcOrv to fr-i—rttL Ur, c^uld 

CEHTUxt 
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Assessing Reading: Sri Lanka 

The illustation on the following page is an example of a task used to assess the reading of 10 year olds 
in Sri Lanka. 

Their achievements are profiled below. 








r.e: 
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An example of the reading task - Sri Lanka 



LET’S GO TO THE FAIR 




Rani : Mother, let’s go 
to the Sunday fair. 

Mother: All right. 

Rohan : Can I come, too ? 

Mother: Oh, yes. Rani, bring the bag, 
please, 

Rohan : Let’s go by bus. 



Profiles of Achievement 
Reading in English, Tamil and Sinhala 



Profile of Reading (English) 




Figure 1 1 shows that 17% of children achieved Level 1, 44.2% achieved Level 2, 30% achieved Level 
3 and 8.8% achieved Level 4. 



Profile of Reading (Tamil) 




Levels of Attainment 



Figure 12 shows that 5.5% of children achieved Level 1, 31.5% achieved Level 2, 37.2% achieved 
Level 3 and 25.8% achieved Level 4. 
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Profile of Reading (Sinhala) 




Levels of Attainment 



Figure 13 shows that 6.5% of children achieved Level 1, 35.5%, Level 2, 43%, Level 3 and 13% 
achieved Level 4. 



Profile of Reading (English, Tamil and Sinhala) 
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Level 1 Level 2 Level 3 Level 4 

Levels of Attainment 



Figure 14 shows the profile of achievement in Reading for English, Tamil and Sinhala 
Examples of Running Records and Teacher comments 
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RESEARCH PROJECT ON ASSESSING THE QUALITY OF LEARNING AND TEACHING - 1996 

(NATIONAL INSTITUTE OF EDUCATION ■ SRI LANKA & BRISTOL UNIVERSITY - UNITED KINGDOM) 









ENGLISH ~ YEAR IV - READING RECORDS 

MntLor- Hnsi-ifux Piuhianiiii Class: ....... 6. SmS.s.Ai 

R* n i * /)nurac(h<x HejrxAtnQj< Class teacher's na me##v. £ N. 

Pohan* ChatnCLta /UaUuSanka School .^h'P. na ^ 

tOkx ■ 

Passage 2: At the fair 


/i ne-bjkki 

ruxtja • 

Scliavl , 

Qiad useli 
a 

i;UU 

\/us>jy c 

i*y ■ 

Urd “ 

f xu it* t 

, 

y^ra n Q 
Po_o *■ 1 

iij CJrrrcJ 
plexC-t • 

7*A*ia A V* 

K/(i V 

1 

y c ru 

0 5 

n 

Ay f >tf* i* cl 
rt/ />Vst 

Excelled 

1 




MOTHER: 


;! ' I': -;' 


; : ;;:; ; warit : 


spiche 




Let r s 






y 




y 


y 

- — < 




g° 


to 


x ; the 




fruit 


steU ] 

i 






y 


y 


y 


i 




first 
















X 










RANI: 














) 




y 


y 


\y 


y 


1 

i 

j 


ROHAN: 


: Yes, 


yes. 




;i; : :x:;;;jwant ' 


a 






v/' 


y 


y 


y 






toy. 


///.; too. 












y 








I 


MOTHER: 




■ A 


11 


If right. 


xhutixx/^x 


X Jet's 


go 






y 


X 


y 


y 




to 


the 




stall 


first 




y 




y 


y 


X 


RANI: 


These 


mangoes 




nice, 


mother. 




y 


\y 


y 


y 


y 




Let's 


buy 


them-: 




46 


i 








. / 
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RESEARCH PROJECT ON ASSESSING THE QUALITY OF LEARNING AND TEACHING - 1996 
(NATIONAL INSTITUTE OF EDUCATION - SRI LANKA & BRISTOL UNIVERSITY - UNITED KINGDOM) 



ENGLISH ~ YEAR IV - READING RECORDS x 

Unde: Class: ..yeCUl.A 

Saman: Arlw'fcl Class teacher's name:. •?> 

Mihiri: IreBhc.Sv.0 School :k/p.d/ilet* f .MS. ^ 



Passage 3: The present 



UNCLE: 




Hhlio,: 




Sah^ahP ; xi 


How 


are 






i/ 






x' 

do 




SAMAN: 




/filled 


thank 


you. 










«/ 






UNCLE: 


Hello,. 


Mihiri! 










;ybu? 






«/ 


t/ 


X 

0 


i/ 


MIHIRI: x 




fine/ 












t/ 


l / 




i/ 


V 


SAMAN;; 






Whatfs 




in 


/ ; : • ;;the:/x 




uncle? 






* ^ 




vX' 


a/ 


UNCIJE:;;;; 








for 






i/ 




P0iSoOt 


>/ 


' V/ 




and 


Mihiri. 










v/ 










MIHIRI: 


■ Oh! 


What 


IS 


it? 






iX' 










UNCLE: 


Gan 


you 


guess? 












g£<? 




i 

H 


SAMAN: 


Is 


it 


a 


toy? 


1 

1 

j 


1 

1 


«/ 

1 




t/ 


X 

j_ 


i 

1 

l 
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5 S 



o 

ERIC 



NQHIP : 


Is ■ :;:S: ; y.|:| 

: -■ 


g:;, ; ;.;.dt,,; : 'v 




book? 






1/ 


1/ 




,, ^ 




SAMAhJ: 




•: /:;: : .;...;:;'we ; .p ;.;. : :;:t;! 


eat . : 












t/ 


EX' 




MltflRI: 






. drink 


it? 






1 


/ ^ 




C^* 




UNCLE: : ; 


: i No, | 




: : : v ';No. 








•if 

✓I 

•jj 




tx" 






SAMAN: 


: ':CWe:::;: > 
■ :•:• .••■ ;• • * 

: . : i i : : : : : :* - r .■;■•; 


x ; ££.&ipi^ 


guess. 








</ ? • 


y 


X 

9 ^cse 








Itfs | 


bijixi.- a . 


Mill 










— 









Uncle. - Pending vJas nice Qisino cmd £a\ling wcra 
in VmS reading But '° So roe places 

t>2 dragged Some v-iofAs and Ssto pped 

readir>q because be C.oo.ldoH read -/^.e 

next word fjeasily / ex <»rr><ple ^'V<e prese t 
Qnd guess V H»‘s pre£entct.oo ua S good • 

Soman - Gu ections U@re not preSe-ndcA proper Oag 

-Boat’s no cap , rising and. ^dUrng o^e-c^ 
no t +bere . 

M ;^ rl - , she woe very fluent in Uer reading. 

«Sbe has no hesitation and no 

m.£ pronunca-f to n . £kp r 2 £S''ng id eoS Oua£> good. 

Rising and. falling k» oas tbere . 

QVeralt presentation ■ rids good A / 1 "the Students 
W0re acti ve, Tbf’tj p^on»aoee(i expressed. 

4f, e ideas nicely. 
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Assessing Writing in Malawi and Sri Lanka 

Writing Tasks 

The writing tasks were designed to gather evidence of children as independent writers The tasks were 
interested in assessing whether children could communicate meaning to a reader and whether they had 
an awareness of wide range of the functional aspects of language, including grammar, punctuation and 
the conventions of spelling and handwriting. 

The tasks took as their starting points, ideas related to passages from books, in some cases those 
related to the reading task (Sri Lanka) or in Malawi, to a text read in class. 

The Literacy Scale and Level Descriptors (Writing) 

The following levels and level descriptions were decided upon by teachers in Malawi. A similar set of 
descriptions for each level was decided upon by teachers in Sri Lanka. It is worth noting again that 
there were differences in what teachers emphasised as being important for each level depending on 
whether the assessments were in English or in the national languages. 

Level 1 

The writing is organised in words and phrases. There is no evidence of use of punctuation or consistent 
orientation in shaping letters. 

Level 2 

The writing communicates meaning in simple words and phrases. The is some awareness of the use of 
full stops. Letters have clear shapes and orientations. 

Level 3 

The writing communicates meaning in narrative or non-narrative forms. Mainly simple sentences are 
used. Ideas are in some sequence. There is a clear awareness of the use of full stops, capitals and some 
with-in sentence punctuation. 

Level 4 

The writing communicates meaning through the use of simple and complex sentences. There is an 
awareness of the reader. Ideas are expressed in a logical way. There is good use of both between 
sentence and within sentence punctuation. Letters are accurately formed and consistent in size. 








O 



Assessing Writing: Malawi 

An example of a task to assess the writing of 10 year olds in Malawi is provided on the following 
page. 

Their achievements are profiled below. 
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ENGLISH - JUNIOR PRIMARY 
WRITING TASKS 



Notes to the Teacher 

These tasks are designed to assess children's capabilities as independent writers. 

What to do 

Task 1 Aural task: extending and adapting 

Read the passage The Hyena and the Hare to the class. 

Ask the children to make up their own ending to the story 

Each child should write in his or her own words on the worksheet provided. 

Task 2 Writing task: short story 

Read the story In the Hospital to the children. 

Provide the children with Pupil Sheet JP Writing 2. 

Ask the children to look at the picture. 

Have a short discussion about the picture and the dialogue you have read. 

Ask the children to write their own version of the story based on the picture. 

Task 3 Factual writing 

Divide the class into groups of two or three. 

Ask the children to collect information and complete the form on Pupil Sheet JP 
Writing 3. 

You should suggest to the pupils how they can go about obtaining the 
information, but you should not give them the information yourself. 



: 3 
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Profiles of Achievement 
Writing in English and Chichewa 



Profile of Writing (English) 




Figure 15 shows that for writing a story in English 23.3% of children achieved Level 1, 30.2% 
achieved Level 2, 40.5% achieved Level 3 and 6% achieved Level 4. 

Profile of Writing (Chichewa) 




Figure 16 shows that for writing a story in Chichewa, 11.5% of children achieved Level 1, 35.5%, 
Level 2, 42.8%, Level 3, and 10.2% achieved Level 4. 
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Profiles of Writing (English and Chichewa) 




Figure 17 shows the profile of achievement for story writing in English and Chichewa. 



O 
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Examples of Story Writing - Malawi 



ENGLISH 

PUPIL SHEET JP WRITING 2 



Child's name 
Class: 

Date: 



At the Hospital 



Dialogue At the hospital 




Doctor 


What’s the girl’s name? 


Mrs Shumba 


Wezi. 


Doctor 


What’s her problem? 


Mrs Shumba 


Last night, she didn’t sleep. She cried and cried. 
She had a headache and a stomachache. 


Doctor 


1 see. How old is she? 


Mrs Shumba 


She’s twelve. 


Doctor 


1 think she’s got malaria. Give her these tablets: 
Four tablets today, four tomorrow and two the 
day after tomorrow. She’ll be all right. 


Mrs Shumba 


Thank you doctor. You said, four tablets today, 
four tomorrow and two the day after? 


Doctor 


Yes, that’s right. 



- 6 # 
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Examples of Story Writing - Malawi 




£NC-ili £H - a-uNiOfk PPomark 

WPM*TiHCr7 T/VS^-S 

TASML: S)/OtX~T STOP-^ 

NAmc: a)\ql cKi'kacwc/c/ 

CLASS : Snrb 5 

bAT£ - 1H 

— - — za?j _ ^&p/ta< 

flOrst) jC CO/7 « 

^ 04 ^ . '*** wWo. ' 

Ifi ^ W 1 J J W- * 

IS , , #0w , o ^ 1 

3i V//?g 1 _ J -Sclc^ xt, r '**? 

j *.•„* *J . *?**«»'”’ *<* ai ° 

. ^ this r „ ^^Oco /W. 

<2^ OnJ dfe / ’ fii'Fbk&n ^ Cr 

/ ^ ™<?(k'e/ 1 ^ ti+iN-y year^g &</ 

Old, f , 

”»*«4e <»x” ^ -V'T* 

rt^O. u 9+ ;il 0 y f ,^. 



knd u# 



o 
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Assessing Writing: Sri Lanka 



An example of a task to assess the writing of ten year olds in Sri Lanka is provided on the following 
page. 



Their achievements are profiled below. 



An example of the writing task - Sri Lanka 



RESEARCH PROJECT ON ASSESSING THE QUALITY OF LEARNING AND TEACHING - 1996 

(NATIONAL INSTITUTE OF EDUCATION - SRI LANKA & BRISTOL UNIVERSITY - UNITED KINGDOM) 
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Profiles of Achievement 
Writing in English, Tamil and Sinhala 



Profile of Writing {English) 




Levels of Attainment 

Figure 18 shows that for story writing in English, 83.6% of the children in Sri Lanka achieved Level 
1, 13.6%, Level 2 and 2.5% achieved Level 3. No students in the sample achieved Level 4. 



Profile of Writing {Tamil) 




Figure 19 shows that for story writing in Tamil,- 18.5% of students achieved Level 1. 37.3% achieved 
Level 2, 38.7% achieved level 3 and 5.5% Level 4. 




ti 
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Profile of Writing (Sinhala) 




Levels of Attainment 



Figure 20 shows that for the writing of stories in Sinhala, 15.3% of children achieved Level 1, 36.7% 
Level 2, 39% Level 3 and 9% achieved Level 4. 



Profile of Writing (English, Tamil and Sinhala) 




□English 

■Tamil 

□Sinhala 



Level 1 



Level 2 



Level 3 



Level 4 



Levels of Attainment 



Figure 21 shows the profile of achievement for story writing in Sri Lanka. 
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An example of the writing task - Sri Lanka 



RESEARCH PROJECT ON ASSESSING THE QUALITY OF LEARNING AND TEACHING - 1996 

(NATIONAL INSTITUTE OF EDUCATION - SRI LANKA & BRISTOL UNIVERSITY - UNITED KINGDOM) 
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ENGLISH 

PUPIL SHEET (WRITING) 



Child's Name: ..S.&d.LMCk. School: 

Class:. RK. Hochai ScW>l i 



Date: !.LJ.L'....?£... 



Eh.i.s. .a £,u.i± kaaoxias. 

,ln....th.e £mi.i sigil^..T.hexe.. are giapes.- in the. 

£)tLu.t siQl.£...Xbeve .axe In ifae £tuii 

sialj. X.b.e.xe cae.....Qap.o.w.s i.n..'..Xhe. .£y..ui.i s.i.d.L 

J.l:..exG. ojss. QtaOqes..... : .in the £iult Stni£ 

.Sam.©. £xiu4. cae iio Ice .o.c.:, e,. . . . LYJ. ., ^ h?.3 .v. is. 

.QvJ.Dg. to. b.uu : jmngQS^.....o.eXai. is ,w.eaii.D.g. 

.Sa.xo.og.* .Uath.e£....has. a bag.v..llo:tb.0.l .is, 



.£a.c.e^......e)y.J 


1 an a 


X 


^>.QL} Cae. \/.exy J 


ne.D.ou., 


.5.e.!,.ex is 


an D.J 


Id 


[ .vaQan* Se.].i.®.Y. 


..is.... 


seJJe.a... 



)S.S..-„ 







&..%..adhAdk..A .All Jktu L* 

A.!:.(-......d^C^-. ^lAjltU a cg-? cx^rre-C ^vi .•^srr.r^j cL.» 

Pvl^cKlaJR. o v\ ,9sy^X......y^?X 

X.vXU. .*/£ .cty^X .W.crrdU: 

L^diJtea JJ> .Cecc.vA.raJ^. - 
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Part 3 - Assessing Mathematics Learning 
General Principles 

Learning Mathematics was assessed in Malawi and Sri Lanka. In the case of Malawi, assessments were 
made in English while in Sri Lanka, the tasks were locally translated so that assessments were made in 
Sinhala and Tamil as appropriate. 

In each case the assessment tasks emerged from the researcher working with local teachers and, in Sri 
Lanka, curriculum officers. 

Tasks to assess Mathematics Learning 

Two areas of the curriculum were selected, one for each of the two rounds of assessment. In the first 
round the focus was on number and operations on numbers. In the second, attention was given to 
measurement. Some tasks were designed to be undertaken by individual pupils who had to provide 
written answers. Other tasks invited pupils to work in small groups which produced written outcomes 
of their work both as a group and individually. Teachers were encouraged to observe the working of 
such group activities and to note particularly the nature of the communication which took place. 

The tasks and instructions to teachers on how they should be administered are contained in the 
appendices. Examples and discussion of these follow. 

The Sample 

Teachers were asked to select all the students from their classes to participate in the research. About 
300 students took part in the mathematics assessments (Junior Primary) in Malawi and about 150 
students in the fourth primary class in Sri Lanka. 

Awarding Levels 

The use of Levels was attempted, to parallel that used in English. However the number of small items 
comprising the overall task made distinctions more difficult. A simple overall level grading was given 
following analysis based on the number of items pupils were successful in answering. Success in 
answering was not simply dependent on the final answer, but on evidence of working where this was 
appropriate. 

The tasks in Mathematics 

In each of the two rounds of testing there was an attempt to use both standard and less common type of 
test item on mathematical content in the mathematics syllabus at the relevant grade level. It was hoped 
that teachers would be encouraged to explore how their pupils responded to different types of question. 

(i) Number 

In the number items there were examples of 

• questions given orally and others which were written, 

• questions involving pure numbers and those where numbers arose in problem situations 

• problems leading to a single answer and those involving some investigation with perhaps 
multiple answers. 

Box II below gives an example of an activity for Infants in Malawi: 
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MATHEMATICS - INFANT 
GROUP TASKS - NUMBER 

NOTES TO TH E TEACH ER 

This task is intended t encourage pupils to use their number skills to explore a simple money situation 
in small groups. We shall be interested in how they work together, what they find out, how they use 
their number skills and how they tell others about their findings. 

What to do 

Discuss this story with the pupils: 

Kodi has two coins © ©. What amounts can he make with these coins? 

[He finds that he could make It, 5t and 6t.] 

He draws pictures to show how he could do this: 

It © 

5t © 

6t © © 

Two friends Mpango and Pezani have four coins © © © © . Work with some friends to see how 
many different amounts you can make. Draw pictures on the Pupil Worksheet to show the coins 
you would use to make each amount. 

Later we will share in the class what we have found. 

Divide the class into small groups of three or four pupils. Each group can work on a shared Pupil 
Worksheet. 

Observe how the pupils work together and he difficulties arise. After they have worked on this problem 
have a class discussion session in which groups report their findings. Encourage them to explain clearly 
and check that other pupils understand. 

Collect the Pupil Worksheets. 

Following the session, make some notes under the following headings: 

• What happened in the groups during the group work sessions? 

• What did the pupils find out? 

• How well were they able to communicate their findings? 

• How were their number skills used? 

• What problems arose? 

We shall discuss these experiences in our workshop at a later stage. 



In the event with widespread participating teachers and schools, together with the only occasional 
contact between researchers and teachers, it became evident that the more subtle elements of feedback 
were being lost through problems of communication and the unfamiliarity teachers had in interpreting 
many of phenomena occurring in the learning of mathematics. A traditional expectation of items being 
right or wrong led to marking and summing of scores in spite of the analysis proposed. It proved too 
demanding to provide the security of the familiar, together with the challenge of that which was in 
form if not in mathematical content new to the mathematical experience of the teachers. Considerably 
more time would have been necessary with the teachers to support them in developing more usefully 
diagnostic testing with their pupils. 

In general pupils in both countries showed themselves relatively competent at the straight computation, 
were more troubled at problems which required the selection of the appropriate operation and often 
required considerable support and guidance in those problems which were of a more open and non 
standard type. In the end the most useful outcome was the variety of items developed with the teachers, 







giving evidence of ways in which the basic number skills can be applied in interesting and less closed 
situations. 

(ii) Measurement 

The selection of activities in this round of assessment was designed to provide data relating to skills 
and concepts but not the application of formulae which are often over emphasised in elementary work 
on measurement. Pupils had to find the length and area of shapes drawn on a grid but also to draw 
shapes according to given specifications. 
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They were also required to measure the size of a drawn rectangle and of the task sheet. 
On dotty paper we can draw triangles using a dot at each corner . 



Use the dotty paper below to draw as many different triangles as you can on a nine dot grid . (Draw 
one triangle on each grid.) 



O 



Write a few sentences about what you have found. 

Finally there was a more investigatory task in drawing different triangles on a 3x3 grid. In the latter 
task teachers were again given guidance on some specific things to observe and several key questions 



were raised with teachers. However the research mechanisms did not provide for the effective collection 
and analysis of this data. It remains to be followed up in a situation where researcher and teachers can 
work more closely together over an extended period. 

The results from Sri Lankan schools in relation to measurement revealed some striking contrasts in the 
performance on particular items and sometimes between schools. Three examples may serve to 
illustrate: 

(i) Students were given an example of a ‘snake’ drawn along the lines of a grid and shown 
how its ‘length’ could be found. They were then asked: 

(a) to find the length of a given snake and then (b) to draw snakes of the lengths indicated. 
In most schools the success rate on these two types of item was not very different but at 
two schools there was a striking difference - nearly 1 00% success rate for the first item, 
only about 30% for the second type which required the inverse approach. Reasons for this 
difference would be worth closer examination but might suggest a very limited range of 
experience with length. 

(ii) Two striking outcomes emerged from the items which required actual measurement: 

(a) in some schools, the level of performance on this skill was very low. This could 
simply reflect the timing of the assessment in relation to delivery of part of the 
curriculum or the availability of measuring instruments in some schools. 
Alternatively, it could arise from a very theoretical treatment of measurement in 
which experience working with the units and tools of measurement were not 
regarded as important 

(b) in a number of schools, including some of those with the strongest results in 
general, there was a marked difference between the success in measuring length of 
lines across the page when compared with those up the page. In some cases the 
success rate was only half for the ‘vertical’ lines. Results for the smaller and larger 
rectangles to be measured were not very different. 

(iii) In both the finding of areas and the drawing of shapes with given areas, pupils’ success 
rate dropped rapidly when the shapes were not rectangular. In attempting to find the area of 
a drawn right angle triangle, the majority apparently adopted a counting squares approach 
but this seldom led to the correct answer. There was little evidence of a holistic view with 
the triangle being seen as half the related rectangle. As a consequence only 25% were 
successful on this item compared with the success rate of about 50% or better on all the 
other items. 

The measurement tasks therefore reveal some interesting aspects of performance which would be worth 
further exploration. The strongest impression is that pupil performance is highly developed over a 
limited range of skills but significant difficulties are revealed (especially in some institutions) when 
problems are even modestly less orthodox. This may reflect teachers perceptions of what is expected of 
them in general, or in particular examinations. However, if the mathematics being learned is to be 
useful to the pupils in the long term current practice could well benefit from review. 
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Chapter 6 - Discussion 

In this chapter we will discuss the following aspects of the research. 

1 . Interpreting the profiles 

2. The reliability and validity of the levels of assessment 
The Oracy Profiles 

Figures 1 and 2 in Chapter 5 show that in Malawi, for English and Chichewa 27% of children and 
40.7% of children respectively achieved level 3. In terms of the arbitrary norms set by teachers 
children could be seen to be under-performing in this learning area. It is encouraging, however, that 
38.8% of children are achieving at least a level 2 in English and if the profile takes the attainment of 
both level 2 and level 3 into account then the overall level of achievement in this domain of literacy is 
promising. 

The verbatim records kept by teachers of children's oral language shows that children have a reasonable 
to good vocabulary in English in this age group. It would appear that the difficulty lies in 
constructing sentences or extended discourse in English. Children at all levels appear to lapse readily 
into Chichewa to convey meaning. A sustained discourse in English without inserting phrases and 
sentences in Chichewa to clarify meaning is rarely achieved. 

In Sri Lanka the oral competencies of children in English is similarly below the norms set by teachers. 
Figure 6 in Chapter 6 reveals that only 20.3 % of children achieved level 3. It is however, worth 
noting that 42.5% of children achieved level 2 illustrating that in this country like in Malawi oral 
.competencies reside more in the knowledge the children have of English words rather than in their 
ability to sustain a discussion. The oral samples collected by teachers illustrate a reasonable 
vocabulary but there is a definite weakness in children's capacity to build up a discussion on the basis 
of what they see in an illustration and to go beyond the illustration to use more imaginative language. 

The Reading Profiles 

The assessment of reading in both countries revealed interesting similarities. On the whole children 
were reasonable to good readers in both Malawi and Sri Lanka if one went by their accuracy scores 
contained in the running records maintained by teachers. The Profiles of Achievement for reading in 
English and in Chichewa in Malawi shows that 32.4% of children achieved level 3 in English and 
39.2% in Chichewa. It is also worth noting that 37.6% of children achieved level 2 in reading in 
English and 17% achieved a level 4. The reading records maintained by teachers showed that children 
understood what they read and were able to comment on the text. Those children who achieved levels 
3 and 4 very often demonstrated (much to the surprise of their teachers), capacities to provide 
interesting endings to passages in reading tasks which stopped short of providing an ending or could 
give alternative outcomes to stories when asked to do so. 

The reading tasks revealed for teachers that students enjoyed reading and had a positive attitude towards 
the activity. In Sri Lanka the reading tasks were conducted in interesting and varied ways. Most tasks 
were constructed in a role play fashion in which children had to engage in dialogue. Readers took on 
the roles of mother, uncle and Rani and showed an acute awareness of how language functions in social 
and cultural contexts such as talking to fruit sellers at the fair. 

The Profiles of Reading in English, Tamil and Sinhala show that teacher norms based on the 
expectation that at least 50% of children would achieve level 3 in each case were not borne out. In 
English the majority of children achieved level 2 (44.2%) while only 30% achieved level 3. In Tamil 
and Sinhala the majority of children achieved level 3 (37.2% and 43% respectively) and a slightly 
smaller number achieved level 2. It is worth noting here that in Tamil 25.8% of children achieved 
level 4. 

The Writing Profiles 

The Profiles of Achievement for writing in English and Chichewa in Malawi are interesting. 40.5% of 
children achieved level 3 for writing stories in English and 42.8% achieved level 3 for story writing in 
Chichewa. The quality of written work, both story writing and descriptive writing in Malawi is high. 
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The samples of writing maintained as evidence by the teachers revealed that children were capable of 
using language imaginatively and meaningfully. This appears to be in contrast to children's use of 
language in the domain of speaking. It is perhaps worth reflecting here that the assessment of oracy is 
not a standard feature in Malawi and the unfamiliarity of the task is likely to have an effect on the 
outcome. 

In Sri Lanka the Profile for writing in English was unsurprising. Over 300 samples of writing were 
analysed. Most children, 83.6% of the sample achieved level 1; 13.6% achieved level 2 and only 2.5% 
achieved level 3. 

As noted in above, English is introduced to children in Sri Lanka in year 3. The pattern of teaching 
English to children in the first instance is to develop their communicative competence in the area of 
speaking and reading. Writing receives less attention in the form of the production of extended text. 
What the writing samples reveal is that children are competent in naming things and spelling words 
correctly and indeed in many cases they are capable of producing simple sentences correctly. However, 
the use of conjunctions and other devices to link sentences has not been widely established at this 
level. 

It is clear from the results therefore that the expectations of teachers of the levels children could attain 
were too high. 

The Maths Profiles 

The assessment of pupils’ work on the mathematics tasks was earned out using an item analysis which 
revealed correct answers, mistakes and pupil responses to the open questions. 

In general, pupils showed a high level of success in the straightforward knowledge and computational 
problems using the four 'rules'. 

Word problems presented greater difficulty but it was often difficult to distinguish between difficulties 
caused by the language (English) as such or the need in word problems to use less algorithmic 
approaches i.e. needing to decide on appropriate strategies and operations to achieve a solution. 

Teachers regarded language as the primary cause of difficulty but evidence from countries where 
English is a first language suggests that the other cause is frequently a problem where mathematical 
activity has become routinised through a narrowly conceived algorithmic approach. 

How reliable and valid were the tasks as a way of collecting data on children’s achievements? 

Ma th em atics T asks 

In Malawi, the infant tasks were prepared in English and it was necessary for teachers to translate the 
instructions and the word problems into the vernacular although words for numbers have been learned 
in English. The group task appeared to present organisational problems with such young pupils in 
large groups. 

Teachers , however, indicated that the tasks were appropriate for the pupils at each level. 

More open questions which had no unique answer presented problems for the teachers in making an 
assessment. For example, Senior primary pupils were asked to 'write down a number which is smaller 
than 10, but which is as close to 10 as you can make if. Teachers were asked to 'mark' some pupil 
scripts before the teacher meetings but not to add marks to achieve an overall 'score'. However most 
did add up the number of right answers, presumably through force of habit. Knowing whether each 
answer is right or wrong becomes important in such a process if one is to achieve a total. Experience of 
what can be learned from the variety of pupil responses and how they might be used as a starting point 
for further teaching is necessary if the potential of such questions is to be realised. 

While most of the time in teacher meetings was spent analysing and discussing pupils' work on the 
tasks, teachers' ways of thinking became apparent. Significant amongst these were 

• personal uncertainties about possible answers to some of the more open questions; as was to be 
expected the range of competence and confidence in mathematics varied considerably within each 
group. 



• ’bounded’ thinking and setting of expectations in relation to mathematical knowledge e.g. 

'children in Standard x do not know about numbers greater than y because it is not on the 
syllabus’. The continuity and underlying patterns in much of mathematics are easily lost, as is 
the desirable opportunity for pupils to extend into areas with which they are unfamiliar through 
appropriate problems and challenges. 

• strongly held beliefs as to why children make certain mistakes and what is needed to rectify them. 
The latter were usually expressed in the form ’they need more practice at this type of question'. 
Beliefs appear frequently to centre on 'practice making perfect'; an awareness of the significance of 
conceptual understanding was seldom evident. 

In Sri Lanka, teachers reported that children had been very eager and highly motivated by the activities. 
In some cases children had turned up for class, although there were habitual absentees. Teachers said 
they also found the tasks extremely revealing and were enthusiastic about their potential. 

Regarding the group work task, it was felt that this did not motivate the less able although sometimes 
the mixture of strong and weak pupils in the group prompted the latter with the right answer but 
without understanding, and sometimes the answer was copied. It was felt it would be better if the task 
were done individually. Some teachers had tried competition between groups with success. In the 
chain question the instructions in the example are not clear. 

The first and second tasks were felt to be easy, but the third, which needed multiplication and addition 
was found to be difficult. However, overall the children enjoyed the tasks done in a small group. 

In Tamil a good deal of time was spent discovering which words children had found difficult or easy to 
read in terms of the number of letters. It was found that words with 5 or 6 letters caused difficulty and 
this needed to be addressed by introducing more longer words into the curriculum. Teachers in all the 
groups were invited to identify individually and collectively examples of good, average, poor and , in 
some cases, very poor, work as basis for building up descriptions of pupils’ achievements in terms of 
agreed criteria of competence. 

Language Tasks 

The language assessment tasks were divided into four areas - reading, writing, listening and speaking, 
although in Year 4 only listening and speaking are in the curriculum. The tasks used did not include 
an assessment of listening. 

Reading 

In Sri Lanka children were given tasks for reading and writing, although these are not in the text 
books. The tasks revealed a surprising amount of knowledge which had not been formally taught. In 
reading, there were three different tasks. In the first one ’Let’s go to the fair’, students appeared to be 
motivated by the presentation of a real conversation and the role play approach. Even the weaker ones 
who habitually do not seek to answer questions or become involved wanted to show what they could 
do. Many of the children were concerned about the marking which took place in front of them, given 
their unfamiliarity with this kind of assessment. Some had to be reassured that it was not a test. 
Having understood the nature of the task, the children both enjoyed it and were able to correct 
themselves. It was felt that this task is easier to read than conventional reading exercises in the text 
book and encouraged the children to use appropriate text patterns. The teachers found various ways of 
implementing the instructions given with the task. One teacher for example gave all the other children 
in the class a different task to do whilst concentrating on the three involved in the dialogue. Another 
teacher repeated the task using different children as a drama. In the event of the course, the subsequent 
participants would have had the benefit of hearing earlier attempts. However, the teachers felt there 
were no real practical problem to using the tasks. For task 2 which is longer, the children apparently 
found it more difficult, and more words were mispronounced such as 'stall' and 'too'. Some children 
did not pause in the appropriate place or were not sure if it was a question. 

In Task 3 there were a number of specific mistakes. 'For' was read a 'off, 'toys’ was pronounced 'tows’, 
'has' was read as 'how', 'the' and ’them’ were new words to the children and they needed help with 
them, and 'guess' was pronounced 'goss' and was also too difficult. It was apparent that schools which 
have a limited English language use, for example Sinhala-medium only using the text book for 
English, found much more difficulty than schools where extra reading material and stimulus material 





generally is supplied with more emphasis on English-medium context. The teachers were very clear 
that from a curriculum point of view there was strong evidence of the desirability of exposing children 
to as much English as possible in all different forms, with as much variety and interest built in. They 
were impressed with how much the children could do, that they would not have expected. 

Speaking 

The children’s ability to speak fluently was affected by the degree to which they were concerned about 
doing the task correctly. This caused some to hesitate, although they were potentially competent. 
Familiarity with the role play built in to the text book helped overcome this problem. The tasks had 
been correctly set up with the setting the scene methodology used in the text book. The children were 
able to produce sentences. Setting was important to the child's subsequent success. The quality of the 
children’s work varied in terms of its creativity, its accuracy, its complexity, its fluency, variety of 
structure, the enjoyment shown by the children, how meaningful it was and the vocabulary used. 
Specific problems were identified such as difficulties in using the word ’the’. An example of the 
benefit of this kind of assessment occurred in one classroom where a teacher gave a child a task to do. 
The teacher said he would not be able to do it, but he was successful, thereby showing the benefit of 
systematic assessment of this kind. 

Writing 

ft was found that the children tended to write the sentences that they had already spoken. The standard 
achieved in writing was regarded as less good than that achieved in the other areas and the teachers felt 
that the curriculum should give more time to it. They identified a number of dimensions of quality 
including spelling, the use of paragraphs, words or phrases or sentences; the use of new vocabulary not 
in the text book, grammatical accuracy, a variety of tenses, the use of linguistic pictures not in the text 
book, punctuation and capital letters. 

How reliable and valid were the Levels of Achievement? 

The question about the reliability and validity of the Levels of Achievement is an important 
consideration for profiling systems. There are a number of methods of constructing a profiling 
system, some which make stronger claims than others to reliability and validity indices. The KEEP 
profiling system in Australia for example does not claim to provide numerical indices of reliability or 
validity and argues that the intention of the system is to profile performance-based assessments in a 
non psychometric fashion. The Victoria English Profiles on the other hand make claim to strong 
reliability and validity indices. Teachers are required to rate student’s achievements with reference to 
indicators specific for each of nine bands. A score of 3 is obtained if all behaviours associated with a 
band are consistently displayed by a student. A 2 is achieved if most of the behaviours are present. A 
1 is achieved if some of the behaviours are developing and a 0 if none of the behaviours have yet been 
observed. This scoring system is then subjected to Guttman reliability estimates (see Rowe and Hill, 
1996:328) as well as test/re-test reliability’s and inter-rater reliability (see Rowe and Hill, 1996: 329). 
The main source of evidence for the reliability and validity of the reading profile is drawn from the 
Victorian 100 schools study (Griffin and Rowe, 1988) in which the profile was completed for 5,000 
pupils. 

Like the KEEP system, the profiling framework used in this study does not make strong claims for 
reliability. The levels were determined by through teachers’ professional knowledge and their analysis 
of samples of pupils work. High inter rater reliability coefficients were obtained. 

Were teachers able to maintain records of achievement for individual children? 

The study found that most of the teachers involved in the study were able to maintain a record of 
achievement for the children they chose to assess. (It is worth noting here that not all children in the 
class were assessed. We shall return to this point later). 

The research established that, on the whole, all teachers involved in the study were able to apply the 
profiling framework in one form or another, in collecting evidence of children’s achievement in literacy. 
Teachers agreed that the descriptions of the levels of achievement provided them with a meta-language 
by which to describe (to children, colleagues and parents) what children were achieving. Teachers also 
submitted that the profiling system gave them a way of assessing domains of literacy which were 
previously either not assessed or assessed in a fairly basic fashion. Teachers agreed that the application 
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of the framework sensitised them to children’s learning in a way that was not possible before. This 
augurs well for the diagnostic purposes of assessment, embodied in profiling systems. 

•Could teacher’s collect evidence? 

The collection of evidence proved to be a new and often 'daunting 5 task for teachers. Most of the 
teachers in Malawi had no history of developing portfolio’s of children’s work and the collection of 
evidence was particularly challenging. Resource limitations proved to be an influence upon the amount 
and range of evidence of achievement which could have been accumulated. The tasks were all 
accompanied by teacher record sheets and pupil work sheets. This allowed teachers to collect samples 
of children's work as evidence of their achievements. Beyond the tasks however, the collection of 
additional information about children's performance was limited. 

Annotating evidence of children's work produced interesting differences between teachers. Teacher's in 
both countries had little prior experience of commenting in detail on pupils work. In both countries, 
the deficit model of learning (right or wrong answers) was strongly held by teachers and a model which 
encouraged them to comment not on the answers but on what children might be doing and the learning 
processes they were experiencing was quite challenging. Teachers in both countries however rose 
remarkably to this challenge and provided substantive annotations of children's work. These 
annotations and comments is evidence that teachers are gaining more of an understanding of the 'how' 
of learning. 

Even those comments and annotations on student's work, which provided less of an insight into the 
learning processes on a particular aspect of work, could be interpreted as giving insights into the 
broader context in which education takes place. Take the following comments by a teacher in Malawi 
who is looking at the relative performance of two students on a mathematics tasks: 

The performance of this child in mathematics is average because wherever he has been given work 
to do he tries as much as possible to get it right. At times he manages to do so but at times with 
difficulties, for example when an example has been given the teacher must repeat it several times 
before he understands what is expected to do. A number of skills and methods must be used in 
order for him to grab a thing. For example, if the teacher is to explain about a triangle using bottle 
tops he/she has to demonstrate, then do practice together with the pupil, then ask him to do by 
himself. In so doing is able to do it correctly. 

The problem in understanding is not because the pupil is not intelligent but because he lacks 
continuous practice at home as well as at school, because of the swollen numbers of pupils in one 
class. Given a chance to learn in a class where the number of pupils is reasonable and the teacher is 
able to do individual help probably he can do better than at present'. 

Of the second child, the teacher has this to say 

According to the performance of this pupil, it shows that the girl is able to 
do a good number of activities with little supervision. For example once she 
has been given an example in Mathematics, she is able to do the rest on her 
own. After thorough investigations it was discovered that she is able to grab 
things fast because before her primary education she had attended nursery 
school. She has got sisters who are in private schools where English is 
fluently spoken. Her family is well to do. According to reliable sources her 
parents once worked outside this country where probably life is a bit 
advanced...’. 

It is clear that the comments have little to do with 'uncovering' the learning processes associated with 
the tasks themselves. Rather, they are comments of a broad nature which draw on the teacher's meta 
theory of the factors associated with educational success. 

Whether the theory is right or wrong is not the concern. Rather, they show that teachers think about the 
factors which influence learning as well as the processes of learning. It is also true to say that these are 
always inter- related. 

How manageable was the system? 
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Another question addressed by the study is to do with the manageability of task administration as a 
method of gathering evidence of children's achievements. According to Sheil and Forde (1995) the 
manageability of a profiling system is vital to its success. They argue that administration of tasks, 
documentation and reporting procedures are time consuming and when demands on teachers become 
excessive, the profiling system may be regarded as unmanageable. 

The study found that administering the learning tasks were demanding of teachers time. Teachers 
found that using the tasks as assessment tools per se was not necessarily new to them, but the practice 
of making notes or recording their findings, constituted a cultural shift. 

The physical size of the classrooms in the schools involved in the study was also a factor. None of the 
teachers involved in the study were able to use the tasks with all the children in their classes. Rather, 
teachers in all schools selected children in their classes (normally between six to ten pupils in a class) 
for inclusion in the study. 

Sampling within a class does not necessarily invalidate the results of a study. It is perfectly reasonable 
to argue that a stratified random sample which is based on teachers' intuitive judgements of 'weak', 
'average' and good’ learners is a useful way to obtain a sample within a class. Monitoring the 
achievements of these learners over time would provide a legitimate basis for generalising about the 
relative performances of larger classes as a whole. 
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Chapter 7 - Conclusions and Recommendations 

The study confirmed that in primary schools in Malawi and Sri Lanka, children’s literacy learning and 
learning of mathematics were not being recorded systematically. Although teacher’s intuitive 
judgments of children’s capabilities were often good, it did not allow them to diagnose specific areas 
in which children were not achieving. Further, it was found that teachers did not have a ‘meta- 
language’ through which to communicate children’s achievements to parents or the educational 
authorities. 

The study set out to address a number of important questions. First, it was designed to consider 
whether a profiling system capable of providing reliable and valid indicators of children’s learning 
could be developed in two countries, Malawi and Sri Lanka. Second, it was concerned to establish 
how far the capabilities of teachers to collect information about children as learners, evidence of their 
learning and to record the achievements of children, could be enhanced. 

7. The findings show that teachers in Malawi and Sri Lanka were able, through a process of 
negotiation and development to establish profiling systems for primary education in which the 
achievements of children in literacy and mathematics could be charted . 

The processes associated with developing the profiling systems, particularly the frequent meetings of 
teachers in each country, produced a number of important benefits which are worth highlighting here: 

• Teachers were given the space to reflect upon their current professional practices and attitudes to 
identify those areas in the curriculum they wanted to change. 

• During the process teachers dealt with theories of child development and literacy acquisition 
accepting some and rejecting others and in the process they developed their own theories about 
the education of young children. 

• Teachers were highly motivated during the research process and they reported that developing and 
refining the profiling system was very satisfying. For many, they felt themselves to be part of a 
collaborative effort for learning about and improving primary education. 

• Teachers were understandably dismayed when students levels of achievement failed to reach the 
desired norms that they had established. That 50% of children failed to achieve a level 3 in every 
aspect of literacy and mathematics assessed was however accepted by teachers and they understood 
that the profiling system gave them a base line for where children were at and what needed to be 
done in order to raise the children’s levels of achievement. Teachers recognized that using a 
profiling system made it possible to demonstrate the many different ways children can work 
towards mutually agreed standards of competence. 

2. Teachers involved in the study were able to collect information about children as learners, 
collect evidence of learning and to record the achievements of children. 

• Teachers reported that they had few problems in using the assessment tasks to collect information 
of children's learning. Some tasks, as discussed in chapter 5, were not as well suited to the 
context of education as others and teachers found them more difficult to administer. But on the 
whole, all tasks were administered and were largely successful in eliciting from students the 
learning processes the researchers were interested in studying. 

• Teachers reported that collecting infomiation of children’s learning through the assessment tasks 
provided taught them a tremendous amount about what children knew and how they leamt. 

• Teachers were also able to collect samples of children’s work and to produce test-like records such 
as running records, and observational notes which provided the actual evidence of children’s 
progress. This was a new and challenging activity for many teachers involved in the study and 
was only possible because dedicated record sheets were provided with every task. 

• The real challenge for teacher-based assessment procedures lie in the collection of evidence of 
children’s work produced over the period of a school term or school year. In the case of Malawi, 
and perhaps less so in Sri Lanka, the level of resources available in schools would make this an 
impossible prospect. However, the study does demonstrate that the collection of evidence is 
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possible even in poorly resourced educational contexts, when record sheets and clear guidelines on 
what constitutes evidence are provided. 

• Teachers demonstrated that they were able to comment on and interpret children’s work samples 
to reach decisions about what they had achieved. 

3. A main question arising out of the research is the credibility of and potential use of the profiling 
system beyond individual classrooms. The issue here is the tension between the need to provide 
data for teachers, students and parents on the one hand and the needs of educational authorities for 
data reduction and for comparisons of schools. The question then was how could the rich 
qualitative evidence based on teacher records that had been generated be summarized to meet the 
needs of the educational system for indicators of ‘quality’ without trivializing or distorting the 
data or diminishing its value for making classroom decisions? Could the data be used effectively 
for these two quite distinct purposes. 

Certainly part of the answer would appear to lie in being able to assign scores or levels to the profiling 
system that can easily be interpreted at both a local, as well as a national level. The goal therefore was 
to provide a robust enough ‘language’ to describe the broad picture of students’ achievement at a given 
stage of schooling (Little and Wolf, 1997, Broadfoot and Gipps, 1997, Dougherty, 1998) for the 
English National Assessment system was designed explicitly to fulfil ‘diagnostic’, ‘formative’, 
‘summative’ and ‘evaluative’ (ie system-wide) assessment purposes. However, it is doubtful whether 
these goals have all been achieved. 

The four point scale was developed in keeping with other international examples of profiling systems. 
However, the research showed that the level descriptors describing each level may need review and 
revision in both countries as teachers become more familiar with a wider range of pupils’ work. In 
many other international examples where teachers have been involved in developing similar profiling 
systems, the development phases were not necessarily longer but more intensive . The levels of 
achievement in Malawi and Sri Lanka were adopted after only two drafts, but other examples show that 
it is often after 4 or 5 drafts that teachers find their anchors and are capable of writing descriptions of 
each level which are clear enough to be used comparably by teachers throughout the system. 

To address the issue of the reliability of teacher ratings, the study provided for the samples of student 
work to be brought to the meetings by teachers, these were exchanged and blind read by their 
colleagues who then assigned a second score. Inter-rater reliability calculated on these scores were 
consistently high. Thus the finding suggests that the kind of methodology adopted in this project to 
generate both pedagogically useful and policy relevant data concerning the quality of learning and 
teaching has considerable potential. 

Moreover, in addition to the high inter-rater reliability coefficients the moderation process itself is an 
important one. Teachers were asked to review samples of work achieved by students who were 
unfamiliar to them, to discuss their interpretations with colleagues an to arrive at a common 
understanding of the relative 'value' of the work. All these processes have been shown to contribute to 
the professional development of teachers (eg Haslen, 1994). 





73 



85 



Recommendations 



1 . It is proposed that the research be adopted as a model for establishing profiling systems for the 
assessment of learning in developing countries. Shorter, more intensive cycles of work would be 
preferable. 

2. The task development process in the research was time consuming and not always suited to 
teachers working on their own. It is recommended that nationally produced standardized tasks be 
developed with the assistance of curriculum developer in each country. 

3. The administration of tasks proved to give teachers little difficulty. Nationally produced tasks 
could be administered by teachers in selected schools 3 times a year. 

4. Large class sizes prevented the administration of tasks to whole classes in areas of assessing 
reading, oracy and in certain mathematics tasks. Sampling within classes for the administration 
of these tasks has proven to be a useful strategy. In such cases information on the learning of 
every individual child is not always possible but information gained of a random sample of 
children is a useful monitoring device. 

5. The processes of gathering, recording and reporting such ‘quality’ information should be 
explicitly ‘low stakes’ and not used publicly in the form of league tables to compare schools, 
regions or countries. If this happens the formative potential for encouraging better teaching and 
learning is likely to be forfeit to more mechanistic approaches to ‘getting the scores up’. 
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